{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mnist手写数字识别\n",
    "\n",
    "  这个作业中，我们使用tensorflow来实现一个简单的手写数字识别的网络，并用这个网络来做个简单的识别示例。\n",
    "  \n",
    "  本作业中，需要参与者应用视频中学到的知识：dropout，learingratedecay，初始化等等，将网络最终在validation数据上的得分尽可能的提高。\n",
    "  \n",
    "  MNIST 数据集的训练集，是由来自 250 个不同人手写的数字构成，其中 50% 是高中学生，50% 来自人口普查局（the Census Bureau）的工作人员，而测试集也是同样比例的手写数字数据。每张手写数字图片都是 28x28 大小，训练集一共55000个样本，validation集一共5000样本，测试集一共10000个样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、 mnist数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-2-f7b8daa96e70>:2: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ./data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ./data/train-labels-idx1-ubyte.gz\n",
      "Extracting ./data/t10k-images-idx3-ubyte.gz\n",
      "Extracting ./data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "(55000, 784)\n",
      "(55000,)\n",
      "(5000, 784)\n",
      "(5000,)\n",
      "(10000, 784)\n",
      "(10000,)\n"
     ]
    }
   ],
   "source": [
    "#导入数据\n",
    "mnist = input_data.read_data_sets(\"./data/\")\n",
    "\n",
    "print(mnist.train.images.shape)\n",
    "print(mnist.train.labels.shape)\n",
    "\n",
    "print(mnist.validation.images.shape)\n",
    "print(mnist.validation.labels.shape)\n",
    "\n",
    "print(mnist.test.images.shape)\n",
    "print(mnist.test.labels.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1、查看train训练集中数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd0AAAHiCAYAAACtERYWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd81EX6wPFnUggloStKr6Ep5RR7QQELYu8N9VAUbJyKenr+PM5T76yA2Cgq9nqo2LCCDQURpUgv0kF6D0l2fn8kzHxnzYbNZnc25fN+vXzdM5nZ3Tm+2Tz7ndmZUVprAQAAiZeS7A4AAFBZkHQBAPCEpAsAgCckXQAAPCHpAgDgCUkXAABPSLoAAHhC0g1QSmml1A6l1P1Rtu+nlNpe+LjWie4fSobrWbHEcD2HFLbXSqm0RPcPJVcZ36OKzTEspZQWkTZa64WF5WNF5OOwZjVE5Dyt9TuRHoeyoYjrWV9E3hORdiKSKiJzROQ2rfV3xT0OZUNx10UpdYWIvCAi12itRwd+3lxElohIutY6z09PEa2irqlSKlVEhojIX0UkS0QWisgJWuvNxT2uvODTXzG01t+ISObeslKqu4iMF5FPktUnlMp2KXgjLxARLSJnish4pdT+/EEuv5RSdUTk7yIyO9l9QVwMEZGjRORIEVkmIh1FZHdSexRHDC+XzBUi8rbWekeyO4KS01rv1lrP01qHRESJSL6I1BGRusntGUrpQREZLiLrk90RlE7hB6hBUjBi8bsuMEtrTdKtbJRS1UXkPBEZm+y+oHSUUjOk4JPz+yIyWmu9LsldQoyUUoeJyKEi8kyy+4K4OFhE8kTkPKXUGqXUfKXU9cnuVDwxvBy9c6Xgk/SkZHcEpaO17qSUqioiZ4tIlWT3B7EpnPt7SkRu1FqHlFLJ7hJKr7GI1BKRbBFpISJtROQLpdR8rfVnSe1ZnHCnG70rRORFzTfPKoTCoebXROROpVTnZPcHMRkoIjO01pOT3RHEza7C//2X1nqX1nqGiLwuIr2T2Ke4IulGQSnVRES6i8iLSe4K4i9dRFomuxOISQ8RObtwGHKNFHz55lGl1Igk9wuxm1H4vxX25obh5ehcLiLfa60XJbsjiJ1S6ggp+J2fIgVLhm4SkQYi8mMy+4WYXSkiVQPl/4nI2yIyJim9QalprRcppb4RkbuVUjdJwQfiC0Xk4uT2LH5IutHpKyIPJ7sTKLUMKfiWa0sRyRWRmSJymtZ6VVJ7hZgE122KiCil9ojIVq31liR1CfFxsRR8cNogIutE5B6t9RfJ7VL8MLzsyhGRaUqp+4I/1Fq301r/6dOzUuoqpdTmwseFPPUR0XOup9Z6kta6s9Y6S2tdV2t9vNb6672NuZ5lXpHvz7201t3DNsa4V0R+LXxchR2uLOf+dE211iu11qdorTO11i211s/urasI71F2pAIAwBPudAEA8ISkCwCAJyRdAAA88frt5V4p5zOBnCSfhd6K+3Y9XM/kScT1FOGaJhPv0Yol0vXkThcAAE9IugAAeELSBQDAE5IuAACekHQBAPCEpAsAgCckXQAAPCHpAgDgCUf7AQDKvpRUE84f1dWpmn3yUyY+/YoBJk77Ylri+1VC3OkCAOAJSRcAAE9IugAAeMKcLgCgzElr1sQpz3+wnomXdB8d1rqKiTa3snH9LxLStVLhThcAAE9IugAAeMLwMiqN1A7ZJp47oI5Tt+Ccp00cEvcI0hSxx2I+tbmFicc+1ttpV2/M5Lj0E6is0lo2N/Fvd9d36v48pGxds/xoEzf4Zr2J8+PXtbjhThcAAE9IugAAeMLwMiqUtCaNnfJv9x5g4tdOfNbEXTNCTrtQ4PNnSNy64GfT/rUXmrjhHa84rZ6bcKyJ81asjL7T+JOUqlVN3PRr5dQ91eg7E6cqe23m7NnptLv15L4mzp+3UFA2qXT7beM5/6xr4iU9Iw8nt/z8r065bf/fTBzavSCOvYs/7nQBAPCEpAsAgCckXQAAPKn0c7qrbznKKavAapGqG2xhUzv3cQdOtl9Grzp+SkL6hugsfuhIE8+99EmnLrj8J7j0JxT2efPDnbVMPGV7y4ivdUiNpSY+N3OrU7dqwiwTf9DRXZKEfQvO46583S7N+qDRK0U1FxGR7rPOMrF61F1ikrHol1L3Ka15UxPnLV1W6ufDn80b0dnES3qOitiu9cQrTdym789OXfi3MMoy7nQBAPCEpAsAgCdlcnh53fXukO/mTrkmHnfSiLi+VvsqUyPW7dZ5Jq6VUs2pW3f5DhOvGu7+Mz62ppeJN1xQ08R5y1fE3E9Edn4vu4QkfDcpd/mP/Yz55OZWTrvPTu5o4uKW+3x3+kUmPuOZp5264HKiD6Rb8Z3Gnyz8pz2YfG63JyO2a/PF1SZuO2CeiUM7ljrt3N+E6Mwf6V639056wsQXvnCLU9f0n9/H8ApYOPQIt9znqUDJvkdbfuYuC8ruP9vEsVzbsoI7XQAAPCHpAgDgCUkXAABPysyc7vxRdi5lbu9hTl2GSg+WPPUo/HVd+6fWCMRu3YvNvjbxZW90N/GmS5o67ViCUAqHHWzC6+rZudUPdx7gNAsu/5m1taGJcwbv57Rb9JC9iNn3VXfq8ufYbeWCy8PSn3UvfG5gomnlHe73Ehr9l/m/cPrIzk7560seDpTsNViW527vmN3PLs0K5e4pdT9yex5i4nG93O+MdAxsUYjY7TnF/n0fd9ZQpy5V2aVizrKgq3512ulQWTwzqOS40wUAwBOSLgAAnpSZ4eWnT3jRxOHDuv/d0MbE6/ZkxfT8/5tmh5CajlfFtIzOih7288pDvV916oI7Fb3cfKKJL3u1u9Nu04X2RByWE5XQlJkm7H/uABOnrt7oNHOX/6wx0co73F2n5hxvl4acOuoapy51jo039LO7X+XqaU674PKkZq/87vYjvP+QtXe4Q8P7p9oh5V3a1vUddKvTrnruj3Htx/a/2ffrwVXcvz3bdY6JW7y1wamrGIOdftS7e4mJO1Wp6tT1mnO6ibPvtdciv4IMJ4fjThcAAE9IugAAeFJmhpeHXnieif/RpaZTt/+7dteZ/A3u8GG0siXyzlOxaD3exqOf6+3UrXnd7pB0fe3lJg4ONYuItO1vh0Wb38Pwcqz0VDvUHO0wbtX17p42I7c0N3GVtdudusVD7DeRX7jcDkMHD1AQEZmWYz/Dcoj9vvXP/jZi3dnzzjdx9XGRh5NVmv0TpqpVi9guXP7Bdnrh8fbPR2zXfdpVJt5/9tyonx+umxp9HrFu61g7zVZ7wWQf3Ukq7nQBAPCEpAsAgCckXQAAPCkzc7p6mj1Bop67EqPMfzU/NMOd63n+8T4mvn7I0+HNjVcvsztv3XXPYfHvWCW060z333FjO/srHpzHrTfTnbftX2upibt84C73OSzDPi64LGhqjvuZ9R/97FKjVHEP2UbJZKXvNvGOsLrckw41cd17lpr4jZafluAVJhX50+/Crul+//G3A15Fs+Uye5rQcVV/MfHRM85x2tV+6QdvfSoLuNMFAMATki4AAJ6UmeFlIB5WXejucjTneDu8H1ziE37YfbAuOJwcXhdcFnT52zc47Vp+VfGXO8TTyFGnO+XrbrOHDbzY0q7Ju+77U5x2Y5rZa5omYaeNlNKV469zym0mV66hz3jafEb4xECBne+7h5Jk6sWJ60RK2O9HGdjlijtdAAA8IekCAOAJw8txsOIu9+zUUNdtUT2uQaodCs078RCnLu3LaeHNEYPgt42DnzHdnxdf13/5iSZe/nd7+AbDyaWzo3H4NbCqKXuO7dhmX4bV2iHDW9fYb6t/NKGb0yr3QPv+WnjSqKj6VP/n0h+GggIH1tla5M+rbYh83WOVc6q99uuvsecvH9RgtdNu23n29ypv9RpJBu50AQDwhKQLAIAnJF0AADyp9HO6aS2bO+WF/Q408VMXjYzqObpXdXcfSlXRfZZpnJZp4pHPD3PqBjY7JqrngKvhG1Wc8vmN7LKUg2quMvF19b532jUKHKAe/ll00YPtTVztqylx6CVERLKf/cMpt8+9PqrHtX7JnjQWmrfIxC3y3Dn2xf85MqrnG7jyaBPXfdX9LoUOb4yI0g5o4JRHtX0lUMqU0kqtXcvEZ01e4NRdmDXcxLVSIp821XHEpSZufC5zugAAVGgkXQAAPKk0w8vbzz/cxH/8xX7W+Nc5rzvtLsraFMOzl/6zS8/PBznlbPmp1M9ZGVV7zx3+zXnPxtMC16l/twFOu2332d1zvjz4DafumH/aXYl+ndbExBxUXzr58xc55RZ3LorQMuxxUT5/2s7olv/8NLqLievnsgwsZunpTrFpWumGlNcNdJdinnXtRBP3r7UqrHXkIeWg/bKK3iXLJ+50AQDwhKQLAIAnJF0AADypUHO6qmtHE9ce4W7/9VFzezJJtEt63t1h5yRm7Wocsd0HD3V3yqk5dqHBFf+yp6X8eR7CqrImPWJdZZHWxP03zlu+ImGvpafOdMqZgYNszp/knn4zrvVHJj7oaruUq+k/mdMty1Qxk795gZnhOvNzPPSm4tPb3O1vR25paOLi/val1q9n4uV/bWvimYOeimPvCmzZVdXE+8f92aPDnS4AAJ6QdAEA8KRcDy//PsT9Svk9F9mlHpdmbXDqluXZkyfm7qlj4htfu9ppV321XWZw4MT1Js7/bX7EftSSyAddL/h7YJeWsCGWJbnbTdz8ve1SGe06054SE1yaIyLywe92uuDAs+Z469OWR5o65dAzdrogt80ub/1A6Vx18YSIdecvtFMIqRN/jtgO0cvfvMUpv7bCnvzTv5Zdu3f0HT867brdZw+xvyDzi7j2acgfHZxyw5vskqG8uL5S9LjTBQDAE5IuAACelOvh5drd1jnl4JByj9/OcOpynzjAxMFdi5pL5B1oot35Jlzo+K4mPqv2mECN+xlnYyiwOf8U99u0FVnwW8oXPvixiX/a2txp53NIObiZ+nn/cYclU4SDzcuD1P32c8ptMhZGbLv+6eYmzpLkbHxf0e1+3h4ek/NwrokfPmB63F8rV9u/1h0m9TNx9t/daca835fH/bVLijtdAAA8IekCAOAJSRcAAE/K9ZxuvX7uMpvWt9iTY1oNdudq02SZlz6JiGzKtrueHF018uea/rMuM3F9ibwkqaL5/RK7JCe4lODx6T2ddq0k/nM/xmEHO8VTn//a9qm2OxcYCnw2TZ8f3Wkm8G/LCa2c8unV7dz8du3uOlV1fa4gsWq+apcA/vhvu+PecVWLar1v+Tpk4kN/usSpq/K2XQba8iX7tz9Zy4KKw50uAACekHQBAPCkXA8v5612v+rfanDZ+Or/hm5FD2rM2bPTKWc9VavIdhVdo6/sxujpN6ea+OYuXzrtxtx4monrzXaHB9O+nFbkc6d2yHbKq3rUN3Hmafb346uDX3DaBZcFhcI+i2Z/fK2Nh3xf5Osi+a4Y8n7EuiW57jVN/7zo3x/41+7by02sZmWZuMXw2U47nW+Hl/ffNjfxHUsQ7nQBAPCEpAsAgCckXQAAPCnXc7plxcmztjrlcbWfDJTsVo9XzL7CaVfn46mJ7FbZFdjy8ugZ55j4y4PfcJpdd+cTJg5JyKkbsu6QIp/6jFqvOeWuGfZxKYHPmOHPF/z82fbt652aDg/brePK4hIEFKiXGvmkrkdWnxz2k82J7Qwi6vD0QKfc/EG7La/Os++wWLfhLeu40wUAwBOSLgAAnjC8HAfn1ZzhlKunZJp4fq49NLn6iNre+lRe1L5mj4mHvO8OGT/QwP675mqnSu7b/xcTh8RWhp8IFFz+szbfHkD/1IajnHafjjjaxG3GuLuZMaRc/u0Jpe67ERLm/pZdTNxE3GV3OrxxBcedLgAAnpB0AQDwhOHlGK0baIcnG6S630Jekmu/RXnxA4NNXP9jd9gSInnLV5j419ObOHWt/1v0N5RFROZ0H23i42ZcYOI/NtaM+JjWQ+1AsZ4606mrJ1ybimxU8w+c8iGP/s3ErW79Ibw5kDDc6QIA4AlJFwAAT0i6AAB4wpxulFRGhlM+9zp7Is620B6nrveUASZu+ixzhdHKW7HSKbe6dGWEliJ9xM731pRFgTiyyrY0obK5+/VLnXK7vo/ZON19/0rIXVoG+MKdLgAAnpB0AQDwhOHlaIXcwcmXxp9g4o9/7e7UNX2TJQiAb83+z53KueX/jozYthVLxJAk3OkCAOAJSRcAAE9IugAAeMKcbpR0rrssqPndzAkBAEqGO10AADwh6QIA4InSmn16AADwgTtdAAA8IekCAOAJSRcAAE9IugAAeELSDVBKaaXUDqXU/VG276eU2l74uNaJ7h9KJobr2bPweoaUUj0T3T+UDO/PiieGazqksL1WSpXLfSZIun/WWWt9996CUmqkUmpe4R/iK4MNtdZjtNaZ3nuIkgi/nicqpX5WSm1VSi1WSvXfW6e1/rzwei5LSk8RDd6fFU/4Ne2ilJqmlNpZ+L9d9tZpre8VkY5J6WWckHT37VcRGSgiPye7IygdpVS6iIwTkWdFpJaIXCgijymlOie1YygN3p8ViFKqioi8JyIvi0gdERkrIu8V/rxCIOnug9b6Sa31FyKyO9l9QanVFZGaIvKSLjBVROaISIfkdgux4v1Z4XSXgu2Jh2qtc7TWw0VEiciJSe1VHJF0UWlordeKyGsicpVSKlUpdaSINBORb5PbMwCFOorIDO3u2jRDyvmQclC5nIgGSuE1ERktIsMKywO01suT2B8AVqaIbAn72RYRyUpCXxKCO11UGkqpdiLyhoj0FZEqUvDp+Xal1GlJ7RiAvbZLwRRQUE0R2ZaEviQESReVyUEiMk9rPUFrHdJazxORD0Xk1CT3C0CB2SLSSSmlAj/rVPjzCoGkuw9KqSpKqapSMJmfrpSqqpTi3618mi4ibQqXDSmlVCsR6SMF34BFOcT7s8KZKCL5InKTUipDKXVD4c+/TF6X4otfzn37VER2ichRIjKyMD4uqT1CTLTWi0TkryIyXES2isgkEXlHRMYks18oFd6fFYjWeo+InCUFU0CbpeD9elbhzysEkq4rR0SmKaXu2/sDrXV3rbUK+2+iiIhS6iql1ObCx4WS02UUo6jr+abW+iCtdZbWurHW+g6tdUhERCnVo/B6NpCCT9soW3h/VjxFXdPpWutDtNbVtNZ/0VpP31unlLpXCkamckSkXJ5Ly3m6AAB4wp0uAACekHQBAPDE6+YYvVLOZyw7ST4LvaX23apkuJ7Jk4jrKcI1TSbeoxVLpOvJnS4AAJ6QdAEA8ISkCwCAJyRdAAA8IekCAOAJSRcAAE9IugAAeELSBQDAE5IuAACekHQBAPCEpAsAgCckXQAAPCHpAgDgiddThsqzJa93csrfHv20iS/pe6NTl/rVz176BCCyRY8eYeKbT/nYqfvo4iNNHJox11ufsA9H2L+zS252D+mZf/xYE7eeeKVT1+qSXxLarXjiThcAAE9IugAAeMLwcpT0shpOud6x1Uy8sW2GU7ffV166hDjKOa2biTdes92pm97tlaie47oVx5r42487O3Utn11s4rzVa2LpIvYhrVFDpzzizOdN3KvaLqdu7OG9TVxvRmL7heKtGXSUiR+44TkTn1Rth9MuV9t42GGvO3XDpV2Rz732xqOccsNX7VRC/oaNJe5rPHCnCwCAJyRdAAA8YXg5SjVWqIh1B1z4u1POfybRvUEsVHoVE89/rKtT9+Hpj5u4dbo7XRCK8vmfafyNfcw1Xzt1XQ7ua+LG5zK8nAiLrm3mlMOHlJE8KsO+pzZd8Ben7uvbHjVxdVVFSmvF3+2Q8tTrhzp1b17f2MTDh57r1O33zORSv3Y0uNMFAMATki4AAJ6QdAEA8IQ53TjYlZfulEs/K4FEmPdEFxPPP/0ppy5Fqpo4JFqi0X95d6c8usmkiG2Hd7FLHB6td7yJk7VsoSJqcvSKZHcBESz+p53Hnd13RFhtdH8xn9nc0sTPvnSaU9dIvjdxTj37LYx0leq0uzRrtYm73fmYU3e53GLiRM7vcqcLAIAnJF0AADxheDlKNU9bHbFuyzvuTjj7ye8RWiLRgsuCRNwh5dl9gsNa7rDT6vydJj5u3G1OXctxe0ycscAu98lfv8Fp1/WNS008rdvLTt3Pu5qbWO/JjdB7lNTuPoeZeFjLJ8Jq0wXJE1wmVKPDphI//uOdWU75ndtPMnGjD78Pb15i2WF/K17/+yMmPrnrINvu2qmlfq0g7nQBAPCEpAsAgCckXQAAPGFOtxj53e3X3Md3fNKp+2WPnRNs8Mospy7abQMRf6uvP9Qpzz89OM9nr9mYLU2ddv+7ppeJ23z3Q8TnzyvmtXNyIs8hjl9pD+eutm1JMc+CkthVz17Tg6swh5tMKs1NJ4v+Zf9+/nZo+DKhogWX4a07153TzVgZ3dxq8w/tdzA6NbvSqZt25BgThy8napFmlw3WnJu43yXudAEA8ISkCwCAJwwvFyM/w34myVTuyTO52u5aFNq2zVufULwB/d9zyiliT4d6cEMHE08+I9tpp5b+EtXzp9asaeIVVx/k1N3e6X8mnr7HnWSodjJDysn0XY57f5G1vLiJAsQip6d7ctdvl0U3pHzzqqNNvPY0O6ybv2FVTP1I/epnEzf9yq0bN+9AE1+QuS6m5y8t7nQBAPCEpAsAgCcMLxdj6dl8Jilv8sM+RwYPL/joge4mzloa+RvKkuJ+qzH/+M4m7jPiCxNfV9sduwoOZZ8276ywJ10Z+fUQs3bXzY6q3dAVvZxylU/iu8tQZbX2Jntg/MAB70b1mOBwsojIkuPteza0s+IfAEJWAQDAE5IuAACekHQBAPCEOd1iZB3AUqCKpPqaPftuJO4crojIxy+PiupxZy/sbeKUc3c6dflRPQNKamCD4Ly6ithu3sdtnHJj+SNBParYUjq3d8r/ucnu8NSj2s7w5kZwp6ngsiCRxM7jqq4dnXLz9J8jtBRZmJtj4lqLE7ekjDtdAAA8IekCAOAJw8uoUBbsauD+oNZSEz734nAT/2dtT6fZxN9bm/iTw4aLq5qJtoR2m7jbh39zWrW71S5fCe3YEW2X4UGzd93hZIb7Y3PsS+7wbHFDykFT3z3YxI02lP4A+mjNG1DdKR+WoSO0FJmww+5YV+29KQnrE3e6AAB4QtIFAMAThpfDpFS1Zyoe0yjyJvWj1h0fKG1PYI9QEnOu7+D+4J0fTXhgqh0mHtbwO6dZSkM75BUKDCeHO+GJwSbOfsgdJuMcZT+CuyC1TQ9eg6pOu5X5gaHPPAaUY7X+2iNNPKDOo2G19iCY1fm7nJpbfre7sjX931oTJ/pKpLVoZuJJpzweVhv5vf3txtaB0vr4diqAO10AADwh6QIA4AlJFwAAT5jTDZNSu5aJn2j4ccR2k761B5i3kmJOrEHC5ZzWzcTLL3J3kkkpZpeioFQV+Pyp3dnZHrPPMXHDh/wtd0CB1Ab7O+Wul8w0cc2UquHNje7jbjNxmwW8R2O1zU6RSmZKRsR2j6w7wX3cscF50cTNkYabd709qD74PY5wmwLL/0RE1gxrZeIazOkCAFD+kXQBAPCE4eUwec0b7LuRiDT9JDfBPUFQSqd2TvmAkfZQ+NFNnjVx8ND6gnLR7lzTzSn/b8qhJn6611inbkzbl03c9wI7ZJn5JkOWXtSv4xRHN/mkyGZbw4YLs5ZwT+HTJ58f6pRbyGR/L67sNJJOje4ht6041SnXePvHCC3ji99KAAA8IekCAOAJSRcAAE+Y0w2z/u7dRf6899wznHKVib+aOPK5FSiN9f3t9nMT7nnEqavlLBWJvCzo1tVHmPjjL+2cU/bj7haf2avtqSKPnHCpUxc8xP6ie+0ysg/edOcakRj5NapE1W5mrnuizAFDWd7l04HfJW+rzS2XHm7iuRc8GdVjvv/O3TLW19JP7nQBAPCEpAsAgCcML4d5+qBXAiX73fNVW2s67RrmrfDUo8pj20VHOOXgkHKtsJ2H5uTaJVuPr+ll4nlDOzrtar37i4lb7rZLGNx9q1ypk351yu3evN7Ev54/1MTjTrrBaZf+6U/FPCtilfXo6qjaDZjuTgs0ltmJ6A4iaHbXXKe8dnx8nz+tcSMTL7i+qVP342XB048i75r12ja7JDT7+U1Ona/Bce50AQDwhKQLAIAnlX54Oa25O0yRpew3HlNVuu/uVGrrO7nfQg4OKY/bUdepe/6C00wc+uU3E2eFfQMxloPlU6q5Q9kd/7LUxBmB34lQWnSHKaDk0po0NnF25rKI7S5d2tPEza5e5dRxbL1fx9Re6JTfbWOni/IXLI7qOVLbtzHxgivqO3VDz3vexCdV2xH2yMhDykFjrz/TxGmzp0X1mHjjThcAAE9IugAAeELSBQDAk0o/p7t7tFvOTrfzefmBw8wz33SXDCHxggfQ3/HVBU5d9i9T4/paqfXrmbj6OHeu9o2WHwVKzOP6sKZ3ExO/v//7Tl2qsvcKm3bbXahS9rhLQFS63clK5+6JdxcrjTaj7ZKtIb27OHX37meX5F1Vc7lTl/q+/fs5c2djiUaXGpNMfGlWdEvFwr2/w+4Ud9vnFzl17X6wy8hi+b5HPHCnCwCAJyRdAAA8qZTDy6nZrUx8a/P3I7a7eInd6ajm634OOK7M6s9wj47YFNpl4qm9hzp13Z4dZOL2//e7ifPXrov4/GmNGpp4R+dGTt2gYa+Z+LTqW5y64DDUk5vt7061b+ZGbIfECU77fNQu8P6d77Zr8/ZAG9/sZzP7iihv8VITTxh+jFM3aIj9dw3fNa5vzZW2EIzjYKd2pwue3GiHvb/+azcTZ/80xWlXFt6j3OkCAOAJSRcAAE9IugAAeFIp53T3NKpl4h7VciK2m/9GWxM30ByInWhZr7vzbse1HmziXwc84dTN7/OMiWefZM8MGrTgwojP/0p7e4JU+PxTcHlS+LzPravtdnZzb7QHX6ttvwoSo+pGexUW5e1y6lqlVSvyMbvC5vmqr+aeIt7qPjfZKf/fgB4mvm6/iU5d+/T4bqMb/D7FS8NOderqjwz2a1ZcXzfe+K0EAMATki4AAJ5UyuHl4ly34lgTN3xtnok5scQQIOiAAAAgAElEQVS/unPtv/ozm1s6dR2qrjBx96p2aPizju8U84xVI9Y8s6WZiR//sI9T1+ae6SZWuxlS9iHzLbtE74IDBjt1v/z9KRP/e307E78z8kSnXaMRTAkl2qJuu018Z+uL3borDzDxyaf8ZOJHD3SnkTq+eIOJVTF/aFu9usHE9X+bHLlhGcedLgAAnpB0AQDwRGmt990qTnqlnO/vxeD4LPRW3HfqT+b1TGve1MQL/lM7YrsH//Kuib/f1trE4ycc7rRrcVf5Gq5KxPUU4T2aTBXtPVrZRbqe3OkCAOAJSRcAAE9IugAAeMKSIZRLeUuXmbjFRcsithspwaVGdpejFlK+5nABVAzc6QIA4AlJFwAAT0i6AAB4QtIFAMATki4AAJ6QdAEA8ISkCwCAJyRdAAA8IekCAOCJ11OGAACozLjTBQDAE5IuAACekHQBAPCEpBuglNJKqR1KqfujbN9PKbW98HGtE90/lEwM17Nn4fUMKaV6Jrp/KJkYrueQwvZaKcWJamVQZfybS9L9s85a67v3FpRSpyulZhVe6O+VUh321mmtx2itM5PTTUQp/HqeqJT6WSm1VSm1WCnVf2+d1vrzwusZ+axAJFv49eyilJqmlNpZ+L9d9tZpre8VkY5J6SVKwlxTpVR9pdR3SqkNSqnNSqnJSqmj9zasCH9zSbrFUEq1EZFXROQ6EaktIuNF5H0+NZdPSql0ERknIs+KSC0RuVBEHlNKdU5qxxATpVQVEXlPRF4WkToiMlZE3iv8Ocqn7SLyVxHZTwqu6X9FZHxF+ptL0i3eySLyjdb6W611nhT8AjQSkeOT2y3EqK6I1BSRl3SBqSIyR0Q6FP8wlFHdRSRNRIZqrXO01sNFRInIiUntFWKmtd6ttZ6ntQ5JwbXMl4LkWze5PYsfkm7xVOF/4eWDktMdlIbWeq2IvCYiVymlUpVSR4pIMxH5Nrk9Q4w6isgM7W42MEMYUi73lFIzRGS3iLwvIqO11uuS3KW4IekW7zMROV4p1b1wyOouEakiItWT2y2Uwmsi8n8ikiMi34jI3Vrr5cntEmKUKSJbwn62RUSyktAXxJHWupMUjEpdIhXsQzFJtxha67kicoWIjBCR1SJSX0R+E5EVyewXYqOUaicib4hIXyn48NRRRG5XSp2W1I4hVtul4A9zUE0R2ZaEviDOCoeaXxOROyvS9y5IuvugtX5ba32Q1rqeiNwrBcORU5PcLcTmIBGZp7WeoLUOaa3niciHInJqkvuF2MwWkU5KqeAUUKfCn6PiSBeRlsnuRLyQdPdBKXVI4fzfflLwrdfxhXfAKH+mi0ibwmVDSinVSkT6iMivSe4XYjNRCr5oc5NSKkMpdUPhz79MXpdQGkqpI5RSxyilqiilqiml7hCRBiLyY7L7Fi8k3X0bJiKbRWRe4f9ek9zuIFZa60VSsBxhuIhsFZFJIvKOiIxJZr8QG631HhE5SwqmCzZLwbU9q/DnKJ8yRORJEdkgIitFpLeInKa1XpXUXsURpwwFKKV2S8EXbIZrre+Jov1VIvK4iFQVkQ5a68UJ7iJKIIbr2UMKknCGiPTWWn+V4C6iBGK4nveKyC1ScD1raK3zE9xFlFBl/JtL0gUAwBOGlwEA8ISkCwCAJyRdAAA88bqJdK+U85lATpLPQm+pfbcqGa5n8iTieopwTZOJ92jFEul6cqcLAIAnJF0AADwh6QIA4AlJFwAAT0i6AAB4QtIFAMATki4AAJ6QdAEA8ISkCwCAJyRdAAA8IekCAOAJSRcAAE+8HngA+JbWrImJNx/eyMSr++xx2g34yyQTD6oz36k76NurTBxaWsPErYf86rQL7dwZuR8HHmDivNVr9tVtoELJ63GIiTd0zHDqdu1vz2TQrXeY+I7Onzrt+tWy75tPdrrPMXhkPxM3fOj70nU2wbjTBQDAE5IuAACeMLyMCmXV4KOc8t1Xv2biszPXRXxcSuDzZ0hCTt2MY8bYwjE27Lz7Zqdds3sjD2tlvJFv4rzjIjbDXsoeRbpuwJFO1YAb3zVx/1qrYnr6kVsamvjdM44wcWjpCqedznWnIRC9LZfZf9cv/zPcxBnKTTshKfrI3xRxj6PN1bZdj2ruVM63Nz1q4qNSbzVx4wfL3lAzd7oAAHhC0gUAwBOGl8OkdG5v4nm3VDPx5V1+dNrdWHeKiXs8OtipO2Bo2RvSqMhSO2SbODicLBJ5SPmP/Byn/HtedRPnS7pTd2gVO8SYGhj2/PXqYU67blvtcPOBj7q/A8fUXWTiCVKzyD5VeimpJlx+9+EmnnndiIgPydF22H5VnntNqwZGJ/dPre7U9atph5H7TXzbxMM2tXbafdHnIBPnLV0WsR/4s61nbTdxurLXNnw4eVneLhPfveKMiM/349yW9vlquMP+3x79tImPOsuuKlj+mPstZ53j/o4kA3e6AAB4QtIFAMATki4AAJ5UyjldlWHH+df0P8Sp+/FOO0+3LWTnDY54/Tan3ddd7NzP8ZdNdermDY1LNxGluXdmmjh8Djd4DU/46RoTNxhW1WmXOvHniM+//lq7ZKXPwK9NfFf9X5x2+e70kePbja0CpT8iN6zEVg6Odh43z8SdX7Xz6C1vn+y0S23fxsRz/57l1M068RkTB5ew3FxnoftiH9jw8+4tnKr89Rsi9hEiza9ZaeKBn9h1crM2HuC0qxNYeZc/f5FEki0bI9Yd/szfTDz/dDu/2+XWG512jR9I/vdtuNMFAMATki4AAJ5UmuHllKp2OHHu0E4mXni6O4z1xGY7JPXWkFNM3OrNsKGrbDtcOKNVF6dOn27XKqTttEsa0r6YVtJuIwr/O/bpQMn9HDnwd7sEoeHZv8X0/PWftdf+y3V2S6q7RvxSVPMizfvE/l41ZnhZRERUmvvnp8rR0Q3XHvQ/O2TYJmxIOSh/zgLbrq9bd2x/O6b50B0jTdy9aq7TLjjc/EXWwe6TMLxcrPxNm0w8fZSdoqm9yF22kz8/8tROtFJ3FH3/2LH3PKe85YFSv1SpcacLAIAnJF0AADwh6QIA4EmFndNNqe5u+7by1WYmXtjNLhd4bFMbp92EG483ceZXP0R8/uBX26tv2urUDZo80cSj19ivym/5Yh+dRkwOrmK3bQzfYm7qfLvMI1tKPweXNcvOx3672112VG92XnhzQ6uIVZVWatPGTnnqIa8V2e6JzS2dcrtn7FxhfnjjKNUfaeeCx11zqIm7N4w8R4zY1RudnH/XPvV/dcqvSOMILf3hThcAAE9IugAAeFKhhpeDQ8pzHz3IqQsOKT+ysa2Jvz6jg9MudUnJv76+/Ep3iLpHtQkm3riffb4Xa3dy2uVv3lLi18KfnTDrXBN/dtCbTt3Y7qNNfL+4S7uildfD7lq23312WqFlmnv96t+6xMQ73nOfQxV9TneltvTChhHrtmu7rOT1B05x6mr9FnnaJxaLr2xu4u/Gu6eJHZ0RMvGC/m5/W95jd1zSeZGnFhB/Oad2c8pX9ppYZLt313UN+0nyl+txpwsAgCckXQAAPKlQw8t/XNrZxAvPeNKp+3Cn3RT/6zM7mjhvydJSv+6eWpHHDufstkNSDCcnRuYg+2v89NvuUH//WvNNPP+pw0zc4b+rnXZrT7Lfajz9hklOXd/a9hCMhmnBUw3cEw5ebDnexH16uxut51VjfFlEJLVeXRPfccWbEdu9vc1+67zWK/EdTg6XP9vuWnTFhP5O3cIz7LTUnL7u35TT3glsc/XTrMR0rhJLrVnTKa+92P7dvnaQO3/Tr+YKEy/N22XiDQ+7h1RUZXgZAIDKg6QLAIAnJF0AADwp13O6aY3cr/DfPvhVE6/M3+nUPXjvQBPXXFz6OaK0ls1N3OfUHyM3RMIFT5N5adipTt2Ae23d3DMDc3Jnus+REvj8GZKQWylFn05/x5ojnfL4r+3ORu1mrnDqrn3InnA04R53rqoyUYHTvi7NWpfEnhSt5tywP4lnFN1ORGTedfb/S/bVCepQBZTSxV2muap7bRNvbWuXXl1ztPvdisH1virmWe2Wbz0/usXE2eOnxNjLxOFOFwAAT0i6AAB4Uq6Hl0P13GG6c2vYjdD/tf5wp67mqyUfUg4esr1y0GFO3Z3XvGHiizKT/zX0ymzXmfbaHHvt1Lg/f7/fe5n4j1uamjhlxkKnXeud9neM/YlK56tN7QKlzUnrB6KXduABTvmKSfaQg5OrrzFxurhDvukqtdSvfcxtdvow+434/w2IJ+50AQDwhKQLAIAn5Xp4uThn1JzulD/of7OJ03dG3h1o42l2N5MPjnrKxK3S3CGRd3fYb9y1fv86py64i83Ujc0CNauK7zSitvEq+83hC2791MSD6swPaxnd58rgEFeHJ93dpJrc/32gZIc6w7/jXJwUVZLWFdfiq5tH1W7W6/Ybrg3k+2JaoqzQddzpvrNrbAyUqiT0tZ0DRUKxnrLsB3e6AAB4QtIFAMATki4AAJ6U6znd0Mx5Tjn7Tfu18fkXPOXUTbnXPSEkGp/sqmfis0b/1alr+tA0E7dru9V9YGAXmwVT7ZxuS+Z0Y5bWrIlTvueusSY+tfo2E4fvJrUx3x6GfsYMew1fPOgFp13rdLvrVNruUnW1SCHN51sRkd3N9iS7C0iU1e7SycOnXWLirvuvNPE3Xx7stKu2VklRdjVwv3vzr3NfN/G5meudut53TTTxR9LdxFmvJ/aEqljwlwAAAE9IugAAeFKuh5dFu8MPrf9mhxIOm3u9UxfqvUmKsnldllNu/o6Nq3xidzZpErZsIfjKesZcp+7f6w8y8WUn2027v789sV+br2hS27Y28YMTXnbq2qbbJT7L8uwQcu+XBzvtWj/1u4nrrrTLifq85P5+zD1xtG13ctg0wOOBHXNiXI4w5tVTTNyYJTCogPI3uX9j9zvDloPHf7SQyRKLl56wuww+8Xx1p+7Lg+0OgZOuaWMr3gzb7aoMLCfiThcAAE9IugAAeELSBQDAk/I9p1uM+s+GzRs8W3S7/ePwWqn16jrlrtXt3PK0nS3i8AqV04J7M00cnMMVEfl8l52L/+f9N5m4+fPudY902k/ry91tQs+ddJqJJ3R8y6k7YqDdQnT/EbHNxzZ+gHncfVmdv9PENZeV/XOaaizkOxo+5a22JxVlnuLW3Tr1GBN/1O5dEx9xzQ1Ouz/lhSTgThcAAE9IugAAeFJhh5d90o3cQerTqm838c3f2NNwsuUnb32qCF444rmIdQ/ffLmJ635Y+iGjRZ+0tAV3REquHjjexO+PqCdIjKwUO4WQU9PG1RL8uqnt7RKTy66ZEPXjmo1dbOKyPxgeH6l16jhlvcfuMBbascN3d4xPvu5q4scvslM5Z1//ldPum2ereutTJNzpAgDgCUkXAABPGF6Og5W96kasS1uf7rEnFUtqYN+vlLDPhxkbcsKbl0rzF+xQ4ct93cMVjq620MQf1s82cf76DXHtQ2WQNTvwjd+T3bpMZQ+dOPJmuxvcnBcT26dGL9gdyG6psyBiu/Zj3V3MWv4xNULLiiWtSWMTd3hvpVP3wXt2+qzpkMR+Q19l2N+PZYMPcepu7/1uePOCPlVZH/aTxkW284k7XQAAPCHpAgDgCUkXAABPmNONg5w6et+NUGIvbzjKxF0bfuvULf2bjVs+2MHEoV9+i+m1dJ49fWRLvnuCSfsq9rPpurPtnG69UdEvVdp20REmLosHa/vS5PWltnBL5HYHV7fn0syRA+Lej8X/sXORbzZ6LFCT4bQbtcXO77d+fKFTl59XORYKbTmskYn/0+B9p+6uq78z8SH1/+bUtR29tcSvtfj82ibOrRNy6u7r+baJL8h0549TRJk4+Kin7jvPaVdLkv/e404XAABPSLoAAHjC8DLKrE8//4st9HWHl2ccM8bEq96zy4ceXdfDaffxN10lGuPOGWri8MMVpufYz6b7vfKrid3Br+Kd949PTTzh9ZoleGTFogO7Fg3b1Nqpu7mOHb69OGuZie9/sbfTru0j9mCE0Iy5Ub3u9vMPd8rTL3vcxNUCS5WCw8kiIu+fa6c48v+IvJyoIquxcpeJ/73+IKfuH/VnmXjeOU85dSnnBId8g8v/lNMuWOc8Psp2IiLrAodlHP3erSbOfts92KQsTARypwsAgCckXQAAPCHpAgDgCXO6CZCq7GeZOrOT2JFyrvXQRSb+8UJ3O83DM3JN3DjNnkPzaNjSokcvdMuRpIh9/lDYbO3H2zrZup07JRaj5hxt4qYyM6bnqAjyN28x8Rd93PlB+cCGwfndBT1GO81eOswuIfrv6+6SkKBLz/nSxrUedeqqqerhzUVE5ImXz3TKjeckdmvDcuGHGSb8+pYjnaqT/m7n5f/X7g2nLritZ/j8bJC73MfOur6yzT297bxMu11nx08GOnXNxtnnaPPhjyYuC3O44bjTBQDAE5IuAACeMLycAPnaDk/WmbO9mJYoTv7adSb+zynnOnXzBu5n4v49vjDxoLqx7UjVb9kJJp46wR32bDlmWaC0QmLR9PzKO6QcSd7SZU751WGBY4duDoR13J2gLs9aY+NrRkT5au5w8gtbG5r4nfOON3HjOT8KIkv7Ypr7A/vWkzNOv9mpWnWxPeB+yrF2OdF58y5y2q3/wJ78owIzOw1fcZeDje1sh/6zv/wp6j6XNdzpAgDgCUkXAABPGF5OgOC3lxEf+fMXOeXWg2z5S6kRiLvF+Ap2c/am4n5jtXJsa598wQMkPn2hvok/b97FaTf3Bvut1mMOs9MJ307pIJG0G7nJKYfmLzGxzp1X8s7iT6qOn+KUW4638UVid/ZKE3da4YCw8l75YeW0LzeWqn9lBdkBAABPSLoAAHhC0gUAwBPmdBNgUa5dJpS62e5gFD5HAaBoOtcuN8lfsNipa3OzLa8N/ryYA8p576Gs4E4XAABPSLoAAHjC8HIcNP/HZKc88B/HBEruUhcAQOXFnS4AAJ6QdAEA8ISkCwCAJyRdAAA8IekCAOAJSRcAAE+U1jrZfQAAoFLgThcAAE9IugAAeELSBQDAE5IuAACekHQDlFJaKbVDKXV/lO37KaW2Fz6udaL7h5LhelYsXM+KJ4ZrOqSwvVZKlcuzA/j2coBSSotIG631wsDPRorI8SLSRkT+qrV+IZrHIfnCr4tSKltEHhaRo0QkVUSmishNWut5xT0OZUMR1/NYEfk4rFkNETlPa/1OpMeh7IjwN7eLiIwRkfYiMkdE+mmtfwnUNxeRJSKSrrXO89rhOOBOd99+FZGBIvJzsjuCUqstIu+LSFsRaSAiU0TkvaT2CDHTWn+jtc7c+5+I9BGR7SLySZK7hhgppapIwXvyZRGpIyJjReS9wp9XCCTdfdBaP6m1/kJEdie7LygdrfUUrfUYrfVGrXWuiDwuIm2VUvWS3TfExRUi8rbWekeyO4KYdZeCI2eHaq1ztNbDRUSJyIlJ7VUckXRRmR0nImu01huS3RGUjlKquoicJwV3Rii/OorIDO3Oe84o/HmFQNJFpaSUaiwiT4rILcnuC+LiXBFZLyKTkt0RlEqmiGwJ+9kWEclKQl8SgqSLSkcptZ+IfCoiT2mtX0t2fxAXV4jIi5pvhpZ320WkZtjPaorItiT0JSFIuqhUlFJ1pCDhvq+1jmqZAso2pVQTKZgLfDHJXUHpzRaRTkopFfhZp8KfVwgk3X1QSlVRSlWVgsn8dKVUVaUU/27lkFKqpohMEJHvtNZ3Jrs/iJvLReR7rfWiZHcEpTZRRPJF5CalVIZS6obCn3+ZvC7FF8lj3z4VkV1SsLZzZGF8XFJ7hFidLSLdROSqwk0T9v7XNNkdQ6n0Fb5AVSForfeIyFlScE03i8hfReSswp9XCCRdV46ITFNK3bf3B1rr7lprFfbfRBERpdRVSqnNhY8LJafLKIZzPbXWYwuvX43g+k6t9TIRrmc58Kf3p4iI1rqd1npMeGOuZ7lQ1N/c6VrrQ7TW1bTWf9FaT99bp5S6Vwr2TsgRkXI5f8+OVAAAeMKdLgAAnpB0AQDwxOspDb1SzmcsO0k+C72l9t2qZLieyZOI6ynCNU0m3qMVS6TryZ0uAACekHQBAPCEpAsAgCckXQAAPCHpAgDgCUkXAABPSLoAAHhC0gUAwBOSLgAAnpB0AQDwhKQLAIAnJF0AADwh6QIA4InXU4bKmwVj/2LieT1HOXUn3jDQxNXH/eitTwBQGaR2bOuUl55Tz8SH9p7l1L3Y7GsT5+r8qJ6/x/UDnHK1d6eUtIsx4U4XAABPSLoAAHjC8HJxtD2DOCQhp2plDxu3GeerQ9grrUUzEy8/u5GJt2XnOe3aZq808fi275s4+4PrnHaNJ9jPnzWnr3Hq9PadJs7/4w8TqzT37bPqpsNMnFfN7W/TR6bZ58vJEQB/tvWSI0x82p0Tnbpx9WZGfFyutu/f8L/VkTw9dJhTHjyvr4nz5yyI6jliwZ0uAACekHQBAPCE4eUYtWq/ysQqI8OpY/gw/tYMOsop/zT4CRNHO5wUbDW/zzNuXZ/Iz/HGtgNN/NzfzjbxqmPdt8/MK9zhqqDTJ15jYvXdL/vqKlBhpVSt6pQX/bOriWdfPsLE0b6vY5WdXsUpz7m5jq27Lrx1/HCnCwCAJyRdAAA8IekCAOAJc7ox+qjduyY+M7OXU5fPnG5cpLZuYeKxNz8eVlvyX91x2/c38bmZ66N+3IVZq208+ikTp4R9Zg3OQE3PcetSt+wusl1ls/YmOze/9dDdxbRMrPQMu7Rs1jHPR2zXp9EhPrpT8Sm7/DI4hysiMvPy4YFS6e8DO7x5Y8S63y54ImLdgye8ZeLnD+tjK6ZEXqoUC+50AQDwhKQLAIAnDC+jzFrV2y7VaV8l8ufDE2deaOIa99WM2C599WYTj2lY26nLqWeXDwx86C2n7uzMdfvurIjM2qNNPPjWgU5d9VkciiEisuMIu7vXnONHRWwXHLqPdelItM8RrHl5a5OYXgt/FjrWDiMv7m9//tuJw4to/Wdvbz/AKf/jW7tcr8n77t+Dau/Zwwpayw8mVl07uk96QeTXC77Ph7esYeKsOJ+DwJ0uAACekHQBAPCEpAsAgCfM6aLMOubyaRHrVufvMvHamQ1MnHpq5Odr8JOdt117aKr7Wj3tsoBo53DDfbC1i4mrj2MOtyhtBi4x8TlZZzt1S65sauKcOnamVWmJSaj+HhPP6flsxHbtPrLz7+1vXxhWuym2F6+MAsuCRMLncUdG9RSnzzvDxKF79nPqsr/7Kfa+lSHc6QIA4AlJFwAATxheRpn14U+dTfzQ6d84dU3TMk0855IREpWrbJiu3OHlXJ0fKLmfRdcHhrKPfes2E088/xGn3V317RB19wuud+oy3/xBIJK/eYstBGMRaXLfiri+1vYL7IHo0tOtW5hrd6Rq//BG279NDCeXRPDEoPCdpqJdGvRjTrqJ9YkrTaxkZVHNyz3udAEA8ISkCwCAJwwvo8zKHmC3gvlLvX5O3cyjXzBxLDsW5YZ9I/b9HfYA62FLejh1KcPqm7jVR3aY+Ngatzjt5p7+pIlX9cp36rLfLHEXUUqr++yJWDdkhd3QPn/+Ih/dqZB0+1Ymdg8uiKz9F9c65VYj7fs3RX6JT8fKMO50AQDwhKQLAIAnJF0AADxhThflQssh7vxc947XR2gZm9o/rTFxtcVLwmrDy/t2cPZyp5wTS6dQKgt6jDZxKOz+YtqUNiZuLRu89amiWdmjlolTirmHG7ejronbjMh1K+N8SHxxgn3887JBG2t3c6049wEAAHhB0gUAwBOGl1Eu5M+e55QzZ8f3+fP23eRP2mZH3jFn5nz3MPRsWROhJRIlJDoQu8vKYj1EobJLa9LYKZ9yyWQTF7d0744vLzRx9pQ4nwpfjBX3uOVgH8OXDV6x1G5bVufD30zsLv4rPe50AQDwhKQLAIAnDC8XJzAGFf7NvPBvvqFyyO15iIkntHXPCJ0c2Li97VM7nTpGMxNv15mHhf0k8nnM+XXtN2gXv2rPQT6k2TKn3aADP7OPEfcrrdc8d4OJm/z7+5J0tdzaeKw7vPzvBuMitu016wITt799ronjPVwbbukbnUz8XJcXon7comfambj21snFtCwd7nQBAPCEpAsAgCckXQAAPGFOtziBbUnCvw4f/Lr5nAdaOXXZ124UVBwpWVkmfmCknccNn9f/erudE9LT47ymqQJKbbC/U952VAsT76pr7wdSzlkf1fON7Tg07CcZEdvOPemZqJ6z3++9TDztkw5OXfPH7Ik4JT/nqnzacMbOfTcqtHxFPRNnby35rm6xur3TpyY+NCPyDHK/ZSc45XqfLDRxIuedudMFAMATki4AAJ4wvBwPVSrL4FLlkFqvrlPe/qrd1L1rRuQdbZ6bdLyJ28iPielcOZd70qEmzrpnqVM3ruUIEweX6BW305Erfd9NCgWHjf+4pWnkhj/MMGFTcZcFVcZ3/V1dPnHKxR1ykN3vp0R3x9j6sZ3i61szuFQscv9+e66jU673R+KWCQVxpwsAgCckXQAAPCHpAgDgCXO6QJjl/do55Z8OGlZku3+v7+SU2z++1sSxnFpUGfx+qv2TM6HlBKfulW2NTLw5v7qJ31vV2Wm37qtGUpTh/Z51yj2q2YUf3X6+2Kmr22d+oLS5+E7DyNfufVr08+2ll1rbfrdi4TPNnLrZnZ6Pqk8d3rzRxK1H+ZnDDcedLgAAnpB0AQDwhOFlVErBXaZERNa+0tDE73R+OKx1FRMN32SHnic8dKzTqtbiH+LXwQqq9hy7y1v2R9c5de0H2yHf/M1bTFxFfnfaNQ4r7/XrJe6Q43FVF8TcTyRf8EQvEZEG99nrOa7pmLDWRd8/fr7LfZ+3HWV3C0z0aUeRcKcLAIAnJF0AADxheBll1ppBR5m4Sk930/tHO7xp4pCO7rPj/UtPM/GQFu86dcGdpoLDyeG+uryOiFwAAAQBSURBVNDuqFRrNsPJJVV/5ORA7NYlcrgv/eW6+26EuNpw9ZEmrjc6um8Kz3/eDik3a7TBqRvV9IsS9+HGj69wym1+S/5OcdzpAgDgCUkXAABPSLoAAHjCnC7KjG0XHuGUfxr8RMS2wQPkc3VuVM//UTs7jxt+AH3wxKAtod1OXY9HBpv4gNnuSTNIrtQG+5u4YXrRS4lERNJ2V8YzgeLvoRknOeW+xzwfoaVIh36zTfzTgfb7Gf0v+shpd33tRSZOV7+YOFeHz/JHvkcMvp+zx95g4jZ/T86uU8XhThcAAE9IugAAeMLwchiVZv9JMmrsSWJPKp/VvdxjAorbuDw4HBzLpuvhB9AHn+P/1vRw6hp9+oeJk7WLDYq27agWJj4788OwWu4p4q3xyHSnPLmbHdY9PMOd5nGW+FwXeblP8N0b7fs6uDOciMio8XbYu+U/fzZx2Nu8TOC3EgAAT0i6AAB4QtIFAMAT5nTDpLRoauJfjnouYrvgspKGH/HPGKvUenZ7vosPmZLEnliPN/zGKX81PtPETxzT3cR5a9YKyo6UsHuI4Hs0bTuz8fGQ9sU0p3zbkAEm/uaB4XF9rRV5OU75obW9TLz8yiZOXYvf7NKgsjiPG8SdLgAAnpB0AQDwhHHRcBs3m/DgF28y8aHHzXWarXi4jYkz303+yRXlVW4He/D4vftPiPvzn/LbeSZeO6mRrVBuuzsvtacWXZi12qk7odp2Ez+REfkEIiRX+BKTF7ccbOL0z6eFN0cc1P/E7ibVtcnNTt30AcNK9dxnD7vdKR/4WHA3uPmleu5k4k4XAABPSLoAAHjC8HKY/A0bTdwisFn2hrB21aRsfNO2vEtfbYfzj5l+qVP3bddXIj5udf4uE/d6yR5I0HrkCqddxio7VNwkN/KG+K8/09XEb1Q/0qnbekhDE2dtLb/DWpXNqDlHm7ipzExiTyqu/LXrTNzk3+ucujP+3a1Uz32gVMzDRbjTBQDAE5IuAACekHQBAPCEOV0kVf7CJSau28etO0OimxNqLnbuPa+YdsX2448/ItZV/325bRfj8yMxVvaMXJf5UWbkSiBJuNMFAMATki4AAJ4wvAyg3ErZbbcWe3TDQU5d3ecnhzcHko47XQAAPCHpAgDgCUkXAABPmNMFUG61uvUHE0+SaknsCRAd7nQBAPCEpAsAgCdKa53sPgAAUClwpwsAgCckXQAAPCHpAgDgCUkXAABPSLoAAHhC0gUAwBOSLgAAnpB0AQDwhKQLAIAnJF0AADwh6QIA4AlJFwAAT0i6AAB4QtIFAMATki4AAJ6QdAEA8ISkCwCAJyRdAAA8IekCAOAJSRcAAE9IugAAeELSBQDAE5IuAACe/D8gDsctzBrBmwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x576 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,8))\n",
    "\n",
    "for idx in range(16):\n",
    "    plt.subplot(4,4, idx+1)\n",
    "    plt.axis('off')\n",
    "    plt.title('[{}]'.format(mnist.train.labels[idx]))\n",
    "    plt.imshow(mnist.train.images[idx].reshape((28,28)))#数据集中1x784的一维向量还原成28x28的二维图片。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、简单神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、计算图\n",
    "#### 1、网络结构及初始化\n",
    "\n",
    "     网络：单隐层网络结构\n",
    "     输入层：每条数据为1*784的1维数组，一次输入1batch数据\n",
    "     隐层：共100个神经元；w：784*100，高斯初始化；b：1*100,0初始化\n",
    "     （激活函数：Relu）\n",
    "     输出层：共10个神经元；w：100*10，高斯初始化；b：1*10,0初始化\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#placeholder方法定义输入，输出，学习率，不用赋初值，主要作用是占位\n",
    "x = tf.placeholder(\"float\", [None, 784])\n",
    "y = tf.placeholder(\"int64\", [None])\n",
    "learning_rate = tf.placeholder(\"float\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "#1、参数初始化\n",
    "##初始化函数：高斯分布\n",
    "def initialize(shape, stddev=0.1):\n",
    "  return tf.truncated_normal(shape, stddev=0.1)\n",
    "\n",
    "###第一个隐层的参数：100个神经元\n",
    "L1_units_count = 100\n",
    "W_1 = tf.Variable(initialize([784, L1_units_count]))\n",
    "b_1 = tf.Variable(initialize([L1_units_count]))\n",
    "logits_1 = tf.matmul(x, W_1) + b_1\n",
    "output_1 = tf.nn.relu(logits_1)\n",
    "\n",
    "###输出层的参数：10个神经元\n",
    "L2_units_count = 10 \n",
    "W_2 = tf.Variable(initialize([L1_units_count, L2_units_count]))\n",
    "b_2 = tf.Variable(initialize([L2_units_count]))\n",
    "logits_2 = tf.matmul(output_1, W_2) + b_2  \n",
    "\n",
    "logits = logits_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2、损失函数与优化器\n",
    "\n",
    "**损失函数：**  \n",
    "\n",
    "    采用softmax+交叉熵损失cross_entropy，loss计算使用了sparse_softmax_cross_entropy_with_logits, 这样做的好处是labels可以不用手动做one_hot省了一些麻烦，该算法会自动将labels进行独热编码。\n",
    "\n",
    "**优化器：**  \n",
    "\n",
    "    在反向传播中，采用了随机梯度下降SGD优化器，其中，学习率为可以根据需要设定。而且，此操作每运行一次，神经网络就完成一趟前向传播+反向传播，根据batch批次来的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "##损失函数：采用softmax+cross_entropy的损失函数\n",
    "cross_entropy_loss = tf.reduce_mean(\n",
    "    tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "##优化器：采用GradientDescentOptimizer（SGD）优化器\n",
    "optimizer = tf.train.GradientDescentOptimizer(\n",
    "    learning_rate=learning_rate).minimize(cross_entropy_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3、性能评估\n",
    "\n",
    "    需要注意的是，由于输出层的logits，是未经softmax函数激活的。所以，输出的数值不是概率分布，若要想看到概率分布，还是需要做一下softmax。\n",
    "    使用正确率评价指标，将输出结果与正确结果进行对比，即可得到此神经网络输出结果的准确率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = tf.nn.softmax(logits)\n",
    "correct_pred = tf.equal(tf.argmax(pred, 1), y) ##将预测的最大值的索引与label值比较\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))##correct_pred元素是bool型，转化成tf.float32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4、保存模型\n",
    "\n",
    "    saver用于保存或恢复训练的模型，主要是保存训练过程中的参数，以便下次训练或测试时使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "trainig_step = 1000\n",
    "\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2、运行网络\n",
    "    以上定义的所有操作，均为计算图，也就是仅仅是定义了网络的结构，实际需要运行的话，还需要创建一个session，并将数据填入网络中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after 100 training steps, the loss is 0.290631, the validation accuracy is 0.8552\n",
      "after 200 training steps, the loss is 0.288441, the validation accuracy is 0.8974\n",
      "after 300 training steps, the loss is 0.232638, the validation accuracy is 0.9188\n",
      "after 400 training steps, the loss is 0.275237, the validation accuracy is 0.923\n",
      "after 500 training steps, the loss is 0.313698, the validation accuracy is 0.9306\n",
      "after 600 training steps, the loss is 0.332043, the validation accuracy is 0.9416\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\saver.py:966: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to delete files with this prefix.\n",
      "after 700 training steps, the loss is 0.162041, the validation accuracy is 0.942\n",
      "after 800 training steps, the loss is 0.24732, the validation accuracy is 0.95\n",
      "after 900 training steps, the loss is 0.0721297, the validation accuracy is 0.9518\n",
      "the training is finish!\n",
      "the test accuarcy is: 0.9498\n"
     ]
    }
   ],
   "source": [
    "# 对于Variable或Queue等资源，当不再使用该session时，需将资源进行释放。其有两种方式：\n",
    "# （1）调用session.close()方法；\n",
    "# （2）用with tf.Session()创建上下文 (Context) 来执行，当上下文退出时自动释放\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    #定义验证集与测试集\n",
    "    validate_data = {\n",
    "        x: mnist.validation.images,\n",
    "        y: mnist.validation.labels,\n",
    "    }\n",
    "    test_data = {x: mnist.test.images, y: mnist.test.labels}\n",
    "\n",
    "    for i in range(trainig_step):\n",
    "        xs, ys = mnist.train.next_batch(batch_size)\n",
    "        _, loss = sess.run(\n",
    "            [optimizer, cross_entropy_loss],\n",
    "            feed_dict={\n",
    "                x: xs,\n",
    "                y: ys,\n",
    "                learning_rate: 0.3\n",
    "            })\n",
    "\n",
    "        #每100次训练打印一次损失值与验证准确率\n",
    "        if i > 0 and i % 100 == 0:\n",
    "            validate_accuracy = sess.run(accuracy, feed_dict=validate_data)\n",
    "            print(\n",
    "                \"after %d training steps, the loss is %g, the validation accuracy is %g\"\n",
    "                % (i, loss, validate_accuracy))\n",
    "            saver.save(sess, './data/model.ckpt', global_step=i)\n",
    "\n",
    "    print(\"the training is finish!\")\n",
    "    #最终的测试准确率\n",
    "    acc = sess.run(accuracy, feed_dict=test_data)\n",
    "    print(\"the test accuarcy is:\", acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\training\\saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "INFO:tensorflow:Restoring parameters from ./data/model.ckpt-900\n",
      "\n",
      "the test accuarcy is: 0.9375\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd0AAAHiCAYAAACtERYWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xm8jdX+B/DPNzNHiUpmJUoqRUkjKg33Xk3qNrjS4Fa3NCkN6JKUxtssKaWobrPS/WlALrqNRCGkKEJChmOO7++Ptc5qPY+999lnD88++5zP+/Xy8n32evZ61nPWXnvtZ61nEFUFERERZd8uuS4AERFRecFOl4iIKCLsdImIiCLCTpeIiCgi7HSJiIgiwk6XiIgoIux0iYiIIpJ3na6IDBSRbSJSKCI1knzP9yKyVURGJ1hHRWSDiNyVudJGT0QmishmEZma67KkorzXr4i0sPu+XUR65ro8mcA6lSp237eJyOBclycVIjLS1seiJNcv9nMsIh1FZIdd79SMFjhCJa3fnHS6toD+v+0i8lgJsnhFVQtUdYPNb1wov60i8k3RyqraDMDdSeTbWlX7eeUcLiLz7Afj4hj7cYOILBeRtSLyrIhU8dKaishHIrJRROaKyEnxNmor7VkRWWfz6+2lNRKRT0VktYg8GHrfeyJyuP+aqp4A4Mok9jUr7L6MEJEfRWS9iHwlIqeVMJtw/fYRkVk2v4Ui0sdfOY367WLzLRSR/4nIgaF92VdE3rXbXSki9xW3ARHpYTuDnt5rF4rIMlv2jt7rzex2K3j7Ml9VCwBMSWJ/IiMivUTkSxHZIiIjU8giXKedbPtYG+uLPBt1aj+bD4nIUhH5TUSGikileBknav8icqKtz2Uicp73ei0RmS4iNb192WLr9MUk9idrRKS2iLxlf6j8KCIXljCL+1S1qZdf3O+tEnyOl9rPxXs2z3oi8o6tIxWRpv7KibZp00+037cb7eerSbwNJ/qOzmb95qTTtX/kAlvQugA2AXgtjfxOC+X5v3Ty88wEcBWA6eEEETkFwK0ATgTQFMC+AO7wVnkZwFcA6gDoB+B1EdkzznYGAmgOoAmATgBulj9++d0G4HkA+wA4s6iTtR+EH1T1y9R3LysqAlgMoAOA3QDcDuDVcOMpIQFwEYDdAZwKoJeInJ9OIUWkOUwjuRJALQBjAbwjIhVtemUAHwKYCGBvAA0BxD3qsu/ZHaa+ZnuvVQRwD4A2AK4B8Lj3lkcB9FbV7ensS0SWAhgM4NkM5bfB5tWnuBWTVVydwrTXwwEcBKAFTJ30T5Bl3PYP4GEAXWA+j096P5yGALhHVdentzdZ8QSArTDfud1gyt0qjfwGIv73Vqp2AHgPQNeSblNE9gDwJsx3Tm0AXwJ4JcG2En1HZ61+S8Pw8jkAViBDv+ztl/txAEalm5eqPqGqEwBsjpHcA8AIVZ2tqr8BuBPAxbYMRQ16gKpuUtU3AHyD+B+kiwDcqaq/qeq3AJ4uygums52oqmsBfAFgXxHZFeYLpG+6+5hpqrpBVQeq6iJV3aGq7wJYCKBtGnnep6rTVfV3VZ0H4G0Ax6RZ1FMATFHVqar6O4B7ATSA+bEAmL//UlX9l92nzar6dTF5DoHpSFd6r9UB8LOqLgMwHubHGUTkHPv6p2nuRyRU9U1VHQNgVYby+1xVRwH4IRP5WcXVaRcAj6rqalX9FaauLk1QxkTtv4aqzlLVmTAdWR0RaQdgH1V9NYP7lBFihvW7ArhdVQtVdSqAdwB0TyPbRN9bKVHVX1R1KMx3XUm3eTaA2ar6mqpuhumgW4vIAeFMkviOzlr9loZOtweAF9S7CbSIrBGRY1PM7yKYhrcwI6WLrxXML+EiMwHUFZE6Nu2H0K+hmfb1AHt0VD9GXkXrzgLQWURqwfxKnwPTwT+sqmsytC9ZIyJ1YY4q/KO/lOtXRATmR9Xs4tYtLiv7L7x8kF1uD2CRmKmLlSIySUQOTlCudjD1MyyU9CtMg20IoDOA2SJSAHOEdVua+1BqpNlmM1YMJK7TWOkNRWS3FLa1QkRai0hrmKOz32COjq5NIa8otACwXVXne6+57xkRaWzrsHEymSXxvZVxSWwz8J1spzK+j1Om4r6js1a/Oe10bQV3gBk+dVS1lv0lloqLAIxMs2jJKACw1lsuimvGSCtKr4mdFYTeH153CEwn81+Y4aFKAA4BMFZEXhKRySLSK9WdyCY7X/YigOdVdW7R62nW70CYz+1zaRbvQwAdxJzMURlm1KAygOo2vSGA82GOhuoD+A+At+26AXboaSiAa1R1h59ml/8B4HUANwH4O4BBAB4DcLCdU3pfRA5CHkuzTjOluDodB+A6EdlTRPbGH1+g1XfOqlhXAngEwHCYo8V/AJgAoKqtz49EpEOiDCKW8DtJVX+ydfhTCfIrymOn/LKkuG2W9Hs30bpZq9+Kxa+SVRcBmJqpo1L7S3tvmC+4ROuNg+nIAOAKVU3lBIdCALt6y0Xx+hhpRemx5gEKvfTN4XVVdTWA82y5dwEwGeYDcSvMUfDFAKaLyERVnZPCfmSFLesomKGZjPwosD8uLgJwnKpuSbBesfWrqnNFpAfMHGs9mPnaOQCW2FU2wXw2x9k8H4A5Om2J4C9twMz7fa2qn8Qqjx2inGDzOQTmiLgPgEUAjgXQCMAzMEfXFEOG6vQumLneGQC2wAxNHgYzvVUiqjoDQEdbtnoAHgRwFMyP4+th5sAni0gTfxQvh0rynZRsfkV57PS9FYuIFHqLB8ZbL41tlvR7N+662azfXA8vX4TQUW6aegB4U1ULE60UOvEq1TMKZwNo7S23BvCLqq6yafv6Z7jZ9J2GRO188LIYecUaPr0cwKeqOgvAwQC+VNWtMHMRpeZIyQ4Bj4A5YaOrqm7LQJ6Xwp64pqpLEq2bbP2q6uuqepCq1gEwAObkjKK5pK8BJNuYTgRwlj2bcjmAowE8KCL+SVNFf5fHYY6w9gBQQVV/tNs8JMltlUuZqFM7d9dLVRuo6r4w89PTNP0T2R4C0F9VN+GPdrkIZlQq3smTUZsPoKI92axIvO+ZYpXwe6voPQXev2SPqEuyzcB3sp3HbhanTEl/RyPD9ZuzTldEjoY5ySETZxlDRKoBOBcZHFoWkcoiUhVm7qeSiFS1R3AA8AKAy0TkQDvX0L9o23beZAaAAfY9Z8F8qb4RZ1MvAOgvIrvbSf+/h/dDRPYCcDXM8CpgTk7qZOcHD0dmT0hJ15MwR4Rd7Ac1LSLSDebykc6qmrH9FJG2IlLBnrH4FICx3jD4aADtReQkO3x8PcwJUt/GyOpimP091P77EuZM9n6h9XoC+Mr+il4FoJqYS1o6oXTV305EpKJtCxUAVLCf65RHykRkF5tfJbMoVWMN3aeQb9w6FZEGIlJfjPYwZ7kOSJBXovZftE5nAFXVnDAImHZ5gpizgqsgQyeepcvOb74JYJCI1BCRYwCcgfROOC32eysV9m9edPllFbuczDbfAnCQiHS17/knzAjUXIQk+x2dlfpV1Zz8g2kQo+KkFcIMIcZKGwhgdIzXLwDwIwApyfu8dAWwX+i1SfZ1/19HL703gF8ArIOZY6zipTW1798EYB6Ak7y0bjBn2RUtV4G5fGKdza93jPK9AOBcb7kRgM9gJvgfDK17MczQaC7qtYn9O2229Vj0r1uq9QvzQd8Wym9YBup3Ksxw0mr7eawRSj8bwAJbL5MAtPLSxgHoG2dbkwD0DL22B8x0wK6hz8FymGHmTsXlkct/9u8bbgsD06jTjjHym5TNOgVwvP1bb7RtslvovYE6RfHtvwrMF3cT77UT7TaWATg/lP9IAINzWIe1AYyBuVzrJwAXemmNbR02jvPencqO5L634n6O7WdgSZx6DfxLdpsATgIwF+Z7dxKApl7aMHjfG0jwHZ3N+s15Y07hg9PffmjWIPQlmeA98+wH6tkE62yGmUi/M9f7mObf50P7pTMh12Vh/aa0/83tvm8EcHGuy8M6zcj+V7H7vgHmEpWclymFfXja1sf3Sa5f7OcY5kfQJrveKbnex6jqV+ybiIiIKMtyfSIVERFRucFOl4iIKCLsdImIiCIS9c0xOIGcO1L8KiXG+sydbNQnwDrNJbbRsiVmffJIl4iIKCLsdImIiCLCTpeIiCgi7HSJiIgiwk6XiIgoIux0iYiIIsJOl4iIKCLsdImIiCIS9c0xiDLuxRf/eKb5hg0bAmnTpk1z8fDhw+Pmcfvtt7v4hBNOCKR17NgxzRISERk80iUiIooIO10iIqKIsNMlIiKKSNQPsefNt3OnTN1M/aqrrnLxU089ldG8DzzwwMDy1KlTXbzbbrtldFtp4AMPSmDlypWB5b322svFr732mou7du0aWZliKFNtNFlbt2518eDBg1181113Bdbzz6148803A2mlqF36+MADIiKiXGKnS0REFBFeMkR5wR9OBpIfUj7ssMNc7A8dfvfdd4H1nn/+eRfPmTMnkPb666+7+LLLLktqu1S6zJs3L7C8yy5/HG80bNgw6uKQZ/369S4eMmSIi/06AoBJkya5+KOPPgqknXnmmdkpXBbwSJeIiCgi7HSJiIgiwuFlKrV++uknFz/zzDNx1zviiCNc/N577wXSqlev7uLKlSu7ePv27YH1FixY4OKPP/44kBY+85Xyz2effRZYrlmzpouPPPLIqItTrm3cuDGw3L179xyVJDd4pEtERBQRdrpEREQRYadLREQUkbye0/30008Dy4888oiLGzRoEEirVq2ai3v06OHi2rVrB9YLL1Pu+HOp4Tun+fO448ePd3FBQUFSeY8cOTKw/MUXX8Rd94wzzkgqTypdli1b5uIBAwYE0m644Yaoi1Ou+Zfd/fvf/w6kffjhhyXO74MPPggs++doHHLIIS5u3rx5ifPONh7pEhERRYSdLhERUUTy+oEH+++/f2A5fJehZIRvlN2+ffu0ylQSTZs2dfFtt90WSGvcuHGmN5fXN1Nfu3ZtYNm//MefOkjWUUcdFVj+/PPP4647e/ZsFx9wwAEl3laW8IEHxfCnn44++uhA2ty5c13cokWLyMpUjLxuo4lUqFDBxeE7TSVrx44dSeXhDym///77gbRGjRqltO0U8YEHREREucROl4iIKCLsdImIiCKS15cMjRkzJrA8Y8YMF7dq1SqQ5s/L+beEe/vttwPr+XMA++yzj4sXLlyYdLkqVvzjz1qvXj0XL168OO57/PldALjllluS3l55kImHVI8aNcrFM2fOjLveySefHFhu1qxZ2tum6PXr18/F++23XyAt3N4o8/72t7+52J+PTdVee+3l4l133TWQ5t/G1X+iVLiew7d/zQUe6RIREUWEnS4REVFE8vqSoUzYvHlzYHnRokUu9oeXf/jhh6Tz9C9n8YeX/fwA4Ndff3XxW2+9FUjLwl2QyuzlCIl89dVXLj7mmGNcvGXLlsB6fj1NnDgxkFaKLinx8ZKhkDVr1gSW/bvL+XcpAoJTUaVIXrfR+fPnB5b97zA/LdlLhvr37x9Y7tKli4v9p0QBwbtaXXfddXHzfOONN1x8+umnJ1WONPCSISIiolxip0tERBSRvD57OROqVq0aWI53x6GWLVumlL9/pnT4Yej+w7PDZ8xSZnzyyScuDg8p+6688koXl9LhZCrG9OnT46ZFfCeicsMf0j/hhBMCab/88ktSefh3kLr00ktdHB4mrlSpUtw8/DOb77nnHhf7D70AgmdUDx8+PJB27rnnuti/g1am8UiXiIgoIux0iYiIIsJOl4iIKCLl/pKhbNiwYYOL/fmK8PyC/xQUf343S/L6coRk+XNCAPDKK6+42L88LPwQ87vvvtvF/iVfpRgvGQq59957A8v+k7u+/PLLQFqbNm0iKVMJ5V0bXbVqlYv33nvvuOv5d6Tq2rVrIG3kyJEurl69etpl8i8LOv/88+OWI3zpkj8H7V9ulgZeMkRERJRL7HSJiIgiUu4vGcoGf7hk+fLlLq5Tp05gvSZNmkRVpDKtsLDQxePGjQuk+UPKdevWdXHfvn0D6+XJkDKF+HeKe+CBBwJpxx13nIvDd6SiaPmXEz399NOBtEwMKftOOukkF3fq1CmQNmHChIxuKxU80iUiIooIO10iIqKIcHg5A77//vvAcu/evWOu598dCUh8th8lz7+TzIoVK+Kud+2117o4Q2cnUo75w4XhO761bt3axf4zrik7Ej0z138gQbb5V+SEn5+bqIx33HGHix955JHMF8zikS4REVFE2OkSERFFhJ0uERFRRDjRkQFjx44NLG/bts3F/nzjvvvuG1mZyrpp06a5eNKkSXHXO/vss10cb66d8pd/pymR4A2A/CfKUHY888wzLk724fTZ5s/zT548OZDmlzFc3gEDBmS3YEXbjWQrRERExE6XiIgoKhxeTpE/hPzWW28F0qpUqeLiIUOGuDibD0Yu6zZt2hRY9m9mv3Xr1rjva9u2rYt516mywb8D2bvvvuti/xIhAGjXrl1kZSqvXnzxxZxsd+PGjYHlJUuWuPi6665LKo969eoFlqP6fuaRLhERUUTY6RIREUWEnS4REVFEOKebohEjRrh4ypQpgbQLL7zQxbxMKDOGDRsWWI73tJDwQ+x5mVDZ8/rrr7t42bJlLr7gggtyURzKgX/961+BZf8Wjom0aNHCxe+8804gbbfddku/YEngkS4REVFE2OkSERFFhMPLSZoxY0Zg+ZprrnFxrVq1AmmDBg2KpEzlSfih8/E89NBDgWVeJlT2hJ/qVaROnToRl4Si5N9hzL8jXUkcccQRLm7evHnaZUoFj3SJiIgiwk6XiIgoIhxeTsC/C1L4zEj/4cjdunULpPGM5dzx71YEpHYTdv+OYkDwTjV+vW/ZsiVuHuE7aCX7UGx/W+Eh9UqVKiWVR1k3atSomK+fddZZEZeE/AfGJ3pA/MyZM+OmnXHGGS5evHhx3PX8/FN9uMILL7yQ0vsyiUe6REREEWGnS0REFBF2ukRERBHhnG6IP2/w5z//2cXz5s0LrNeyZUsXJ3s3FMq+Bg0apJ3HlVdeGViuX7++i5cvX+7ioUOHpr2tRML70rNnz6xur7T67rvvAss///xzjkpCYf369XNx9+7d467Xpk0bFyeaj012rjbZ9fr375/UelHikS4REVFE2OkSERFFhMPLIatXr3bxpEmT4q7nX7ZQu3btbBaJsPNlWc8991zWthV+uEKyKlb8ozkleiD2xRdf7OKjjjoq7nrHHHNMSuUoa954443Asn/Z1nHHHedi/2b2FI3TTjvNxeGHwvsPo8i08LaOPPJIFz/11FMurlmzZtbKkCoe6RIREUWEnS4REVFE2OkSERFFpNzP6a5duzaw3L59+5jrjR49OrB82GGHZa1MtLNnnnkmsHz88ce7eOvWrUnl4d+KriSX+/Tp08fF++23X9z1Tj/9dBfvtddeSedPO9u2bZuLX3nllbjr9ejRw8Wp3hqQUuc/+H3ChAmBtNdff93Fmb5059FHHw0sn3nmmRnNP5v4KSUiIooIO10iIqKIiP+UiAhEurFkPPzww4Hl3r17x1xv0aJFgeXGjRtnq0jZIlnIs9TVZzmSjfoESkmd+pcF/eUvfwmk+ZeL+JeHlIGnMJXZNvr111+7ODw0/Pzzz7vYv5zu2muvDazn91VNmjQJpPnD3KVIzPrkkS4REVFE2OkSERFFpFwOL/s3UD/iiCMCaevWrYv5Hg4vx1Qq6rOcKtPDy+UU22jZwuFlIiKiXGKnS0REFBF2ukRERBEpl3ekmjJliovjzeECwQfVV6tWLatlIiKiso9HukRERBFhp0tERBSRcjm8nMjRRx/t4g8//NDFHF4mIqJ08UiXiIgoIux0iYiIIsJOl4iIKCLl8jaQ5RRvMVe28DaQZQ/baNnC20ASERHlEjtdIiKiiEQ9vExERFRu8UiXiIgoIux0iYiIIsJOl4iIKCLsdImIiCKS952uiIwUka0isijJ9VuISKGIbBeRnnHW6SgiO+x6p2a0wBkmIlVsObeJyOBclyddIjLQ7kuhiNRI8j3f28/A6ATrqIhsEJG7MlfazCtr9QmwjYrISbacO0TkpFyXJ11so+m10VLV6YpIcxHZnKhi4rhPVZvGyK+2iPwqIlOLXlPV+apaAGBKeP2QpapaoKrvefldIyILRWSdiHwpIsd6aVVEZJiI/CIiq0VkrIg0iJe5iAwXkXm2IV4cSjvRbmeZiJznvV5LRKaLSE1vf7bY/XmxmP2JjIj0sn+fLSIyMoUsXrF/+w02v04i8pGIrI31xa2qzQDcnUS+rVW1n1fOLiIyyzag/4nIgV5aFRF5SESWishvIjJURCrFylREjrN5+P9URLra9LyuT19pbqNi9BORn2wb/beI7Optq4GIvG3b5xIRuTLBfvYN1ecm21b3sOl9RGSl/fwc5L3vGBEZ4+elquPt/vxUzP5ERkRaishE26YWiMhZJcwi3EZricjzIrLC/hvor5ylNjosVEdbRGR9nP0tNW20VHW6AJ4A8EUG87sXwLeZyEhEjgRwD4BzAOwGYASAt0Skgl3lOgBHATgEQH0AawA8liDLmQCuAjA9RtrDALoAOBXAk942hgC4R1VjfrBKkaUABgN4NkP5bbB59clQfhCR5jCN5koAtQCMBfCOiBQ9eetWAIcDOAhACwBtAPSPlZeqTrFfQAW2Mf4FQCGAoh9s+V6fvlLbRgFcBKA7gGNg2mA1BNvgaAALAdQF8GcAd4tIp1gZqerdoTq9F8AkVV0pIvUAXAZgXwDDYL4XYD87DwK4PkP7kxW2nG8DeBdAbQCXAxgtIi3SyPYhANUBNAXQDkB3EbkkzXImbKOqemWojl4G8FqsvEpTGy01na6InA/TUU3IUH5HwXxhPpeJ/GA+TLNVdZqai5tfALAHgL1s+j4A3lfVX1R1M4B/A2gVLzNVfUJVJwDYHCO5hqrOUtWZALYCqCMi7QDso6qvZmh/skZV31TVMQBWZSi/z1V1FIAfMpGfdQqAKao6VVV/h/lSbQCgg03vAuBRVV2tqr8CeBTApUnm3QPA60VHAcjz+iySB220C4ARqrpYVQth6vQ8EakuIgUAOgK4S1W32bp4HUnUqYgITGf+vH2pMYCvVHUdgPEwnS9gOtt3VHVRhvYnWw6A+VHykKpuV9WJAD6G2cdUdYEZzdho938Ekm8v8RTXRh0xw9xd8UcdFSdnbbRUdLp2CGgQgBtjpDUWkTUi0rgE+VWA+UXeC5m79+g4ABVE5Eib/6UAZgBYbtNHADhGROqLSHUA3ex7UrFCRFqLSGsAOwD8BvNL7Nq09qCUsPV5bPFrZrcYCN4btWj5oATpDUVkt4SZmro/B8HGn/f1mSdtNFadVQHQ3Hs9nH4QincczNHxG3Z5AYCDRaQWgJMAzBaRRgDOB/BAyqWPTqx7Agf+Fim20VT+tsXll6iN+roC+BXA5GIzzXEbLRWdLoA7YX+hhhNU9SdVraWqJZkPuRbAZ6o6LWMlBNbDNLqpALYAGADgcv3jll7zYeZsfgawDkBLmC+pVFwJ4BEAw2F+ff4D5uiiqoi8L2Z+c6dfe/nC1ufU4tfMqg8BdBBzQk5lAH0BVIYZIgPMD6brRGRPEdkbfzS+6jtnFdAVwEoA//VeKwv1mQ9tdByAniLS1P44usW+Xt0OD34M4HYRqSoibWDqqrj6BP44KioEAFVdBeAuABNhhqlvgqnfWwCcJSL/tXPHDTO4b5k0F8AKAH1EpJKInAxz9Oj+Fim00fcA3CoiNUVkP5iDkmT+tokU10Z9PQC84H0fJ5LTNlqx+FWyS0QOhfm1eFiG8qsP06DbluA9hd7igXFW6wnzQWoF80v3ZADvishhqroUwJMAqgKoAzMHeTPMl8CRJd0HVZ0BMxQGO3/0IMx88X9hhrCWApgsIk2S/JCVKyIyDuboBACuUNWdTnhQ1bki0gPA4wDqwcz3zQGwxK5yF8w80gyYH1lPw3xGVxSz+Z0af77XZx610WcBNAIwCea77UGYYc+iOu0Gc3S9GGaq4sUEeRVttxqAcwGc4b+uqi/DzCFCRP4M8xn5CuZcjVYAToc56j0/id2LlKpuE5EzYea7bwHwJYBXYfYhVdfa/L6DmVZ6GcAF8VbOUBstyqsRzI+GvydZ1py20Zx3ujA72hTAT2bqBAUww7gHqmqbFPJrB1NBc2x+1QBUE5HlABqo6vbwG+zEuiMi+4bXAdAawFhVnW+X3xORZQCOhpkbag2gn6qutnk8BmCQiOyhqitT2I8iDwHor6qbRORgAF+q6lYxZ9LuieI7gXJHVU9Lcr3XYeoOdqjwUtiThFR1E8zQZy+bfjmAabE+P0Vs4+8I4IoEm83H+uyIPGijqroDZgRqgF3nZJiRp59t+o8wJ9AU5fESgM+LKevZAFbDdOQ7sZ3y3QBOgxnGXqyq60TkC5gjs1JJVb+GNzcqIv9D8vOhsfJbDfOjpii/u5Hgb5uJNuq5CMD/VLXYcz5KQxstDZ3ucJiTjorcBNPA/5FifuPs+4ucB+BCAGck+sJMwhcA+tnOdCHML/8WAGZ56ReJyCQAG2HOTF4ar8O1wyW7wMxRVBKRqgC22i+OonU6A6iqqu/alxYCOEFEFsPMVWXkRKVMs2cXVgRQAebLuSqA3+3JEKnktwvMsFIlsyhVAexQ1a1plrMtzJFsbZhf02NVda5NawAz17gMZrTidpgzVhPpDtP4v4+zvbysT+RJGxWR2gB2hzmKbQngXwAGFbUpEWkJc5S0BcBfYUarWhaTbXHDlv0BjFTVpSKiAPYXkboAOiGzJ/5llIgcAjMltgvMd1U9ACPTyK8ZzEl2a2D+rpcjxglPKeQbt416LoI5ySoZOW+jOZ/TtWe7LS/6B3Ma92Z7xmjRSRqFyZ6koeYaKj+/tQC22TgdL8B88UyCmbN9FGZYpOgDcBPMmcjfwUzo/wmAu/ZNRMaJiP/L9wMAm2COlIfb+Hhv/SoA7oe5FKnINTCXKIwHcFWaPyKyqT/M/twK4G82dpfb2Po8Ls57Yzne5vF/MGeOboL5+6XrEZgviXn2f394qhmA/8FMFTwP4FZVdduMUZ+AafzDjM3+AAAgAElEQVQxjxbyuT7zqI3uAfMZ2QDTsT+rqsO99FNgOsLfYObwTi3aB7sfgc+l/eF1Akzb34mI7A/TwTxm92sZzOVDs2GGW29Lc3+yqTvMD8oVAE4E0FlV3fByCm20LYBvYM59GQKgm6rOzkA5E7XRojPgGyLGpUKlto2qal7/g5lrKwTwfZLrN4epvI0ALo6zTtGX/BoAp+R6H4vZnyq2nBsADMh1eTKwP/3tvqyBOY0/mffMs5+BZxOssxnmy/3OXO9jeapPu0/lvY2eaMu5CUCnXJcnA/vDNppGG+XzdImIiCKS8+FlIiKi8oKdLhERUUSiPnuZY9m5E+suNOlifeZONuoTYJ3mEtto2RKzPnmkS0REFBF2ukRERBFhp0tERBQRdrpEREQRYadLREQUEXa6REREEWGnS0REFBF2ukRERBFhp0tERBQRdrpEREQRYadLREQUEXa6REREEWGnS0REFBF2ukRERBFhp0tERBQRdrpEREQRifoh9qXC4sWLXdypU6dA2oIFC7K23VmzZgWWGzdu7OJdd901a9ulkpk+fXpguW3bti5+6623XHz66acH1ttlF/6GTceGDRtc/Le//c3Fxx9/fGC9Sy65xMW1atXKfsGszZs3B5bnzJnj4tatW7u4QoUKkZWJ8g+/JYiIiCLCTpeIiCgi5XJ4+cMPP3RxeMgom15//fXA8q+//uriJ554IrJy0M42bdrk4rPPPjvuemeddZaLt27dGkjj8HLJhNtes2bNXLx69WoX16tXL7BeroaU27RpE0hbtmyZi/1pqTp16mS/YHlsy5YtgeW7777bxTNnznTxG2+8EVivrAzb81uCiIgoIux0iYiIIsJOl4iIKCLlZk53x44dLvYv+4jScccdF1ju16+fi8Pzg5UrV46kTGR88803Lv7xxx/jrterVy8XV6xYbppPxmzcuNHFPXr0CKT55zj885//dPGAAQOyX7A4Hn30URfPmzcvkPaf//zHxZzHTWzy5MkuvvTSSwNpCxcujPme8HditWrVMl+wHOCRLhERUUTY6RIREUVEVDXK7UW6Md/s2bNd7N895v777w+sd8MNN2StDC+//HJguXv37i5et25dIK169eqZ3rxkOkPksD7T9fvvvweWO3fu7OJJkybFfd+MGTNc7H+OciAb9QlkuU79u7Il+vutX7/exVloC3EtX748sFy/fn0XX3bZZYG0xx9/3MVVqlTJxObLVBv1v9NatGjh4hUrVgTWE4m921dddVVg+b777nNxngw1x9wxHukSERFFhJ0uERFRRNjpEhERRaTMXvPg36INAE444QQXH3jggS6++uqrIyvTq6++Gtm2KLGff/45sJxoHte/NCjH87h5x39yELDzeQ2+Dz74wMW5msc9/PDD467nP/kIyNg8bpnlX27lXw6WrKFDhwaW/c+OnzcAnHfeeS4u7beL5JEuERFRRNjpEhERRaTMDi8PHjw4sOxfgvD555+7ONt3fvKfXjNmzJhAGp9KkzvhJ5gkcv7552exJGWbf9c1AHjkkUdc3LFjx0Ba+I5tUfniiy9cvHTp0kDajTfe6OIOHTpEVqZ8tHbt2sDygw8+GHO9o446KrDcuHFjFyeagvvtt99cfPPNNwfSTj/9dBcXFBQUX9gc4rc+ERFRRNjpEhERRaRMDS9/+umnLn7xxRcDaQcffLCLmzRpElmZ/OG08HCy/7B0ngkZrfHjx8dNC0853HPPPdkuTpkVvtuQ3wbC7TCbZ51u27YtsDxs2DAXDxo0yMXh8obvWEfxfffdd4Flf7jZH/4NT7P5d4fzH4Zw/fXXB9b79ttvXRy++uCss85y8TvvvOPi0njnKh7pEhERRYSdLhERUUTY6RIREUWkTM3pvvDCCy4uLCwMpPXt2zeycqxZs8bFjz32mIvDc1Z33nln3DTKvB9++MHF48aNi7tezZo1A8sNGjTIWpnKs1GjRgWW/cv6atWq5eJUn/zlz9v7d7sCgPfffz/me6644oqUtkU7P3Tenx8PXzrm8+/45j/t67DDDgusN3fu3Lh57Lrrri4u7d+lPNIlIiKKCDtdIiKiiOT18PLmzZsDy/GGjADgjDPOyHZxnOeee87Fv/zyi4vbtm0bWO+AAw6IrEwETJs2Lan1+vfvn+WSlB+9e/cOLL/11lsuXrx4cSDNv5RE9Y9nr48cOTKlbft5xHtQOhBsh+E72VHyRowYETfNvwPcEUcckVR+EyZMSHrbxxxzjIuzfZfBdPFIl4iIKCLsdImIiCKS18PL27dvDyz/+OOPLo7yOblh4TuzFEl2WIWyY+rUqXHTateu7WL/rjiUnkaNGgWW/TNQFy1aFEh79913Xezf0H7vvfcOrJfs2cz+828TnYF+yimnuNj/HFDJXHbZZYFlf1pgypQpLg4/W3fBggUufumll1y8cuXKwHp+3axatSqQdu+997q4W7duLq5bt24yRY8Uj3SJiIgiwk6XiIgoIux0iYiIIiL+afURyOjGwk8OOfnkk10cvpxo4sSJLs70kyc2bNgQWPbvjuILPzj9zDPPzGg5ihH/monURfrhSYU/X7T//vu7eMeOHYH1mjVrFvM9pVg26hPIgzpNlv/Q8zp16gTSjj32WBe/9957Lq5evXr2CxZfXrfRTZs2BZb9+Xy/LsJ9TrzLuf76178Glp944gkXn3DCCYG0b775xsW33Xabi3N8CVjMHeORLhERUUTY6RIREUUkry8ZqlSpUmC5ZcuWLh4+fHggzX/I8YABA0q8renTpweW58+f72L/RvpA/OGSRHfFoezwHz4RHlL2nXPOOVEUhyLkDy2G296TTz7p4hwPKZcZ4Wm7yZMnu9gfzveHmsPuuOMOF/vDxEDwwQg9evQIpN14440ufvXVV10cviNaabgkjEe6REREEWGnS0REFBF2ukRERBHJ60uGwlasWOHi8Lyt/8Ds8OVEyQjfTsyfI/KfJATsfEp8kfAp9eE56SzL68sRUnXddde5+NFHH3VxeG7Hn7Nv0qRJ9guWPl4yFPLJJ58Elv0nz+y2226BtK+//trF4VtV5lCZbaNz5sxxsf8UNiDYFv252URPCwpfLnrllVe62L/9pN/+AeBf//pXcgXODF4yRERElEvsdImIiCJSpoaXE1myZEnMOFnt27ePmxY+Ld0fxvT9/vvvJd5uBpXZoSvfunXrAsu77767i/1Lhtq1axdY77PPPstuwTKPw8shffr0CSw/+OCDLu7Vq1cgLV4bzbFy0UazwX+CWIcOHVwcniqaPXu2izN9Z8IYOLxMRESUS+x0iYiIIpLXd6QqiYYNG8aMM6F58+ZJrbds2bLAcr169TJaDgJmzZoVWI53Fyr/QddUNrz88suB5Ro1arg4PPRMZYt/pvpVV13l4qFDhwbWe/75513sn/EcJR7pEhERRYSdLhERUUTY6RIREUWk3MzpZlP4sqt4l2FxDjf7Vq5cGTfNv6tYz549oygOZdnYsWNdvHTp0kCa395K0V2nKAv8OwTeeuutLg7f/erqq692cfjJYnvssUeWShfEI10iIqKIsNMlIiKKCIeXMyD8gGw+rD53xowZEzdt//33d3GVKlWiKA5l2T333OPicLtLdFnYli1bXOw/ACX8YATKPw0aNHDx8OHDA2l/+9vfXNy3b99A2hNPPOHibD6Mhke6REREEWGnS0REFBF2ukRERBHhnG4GhB9O74vgSRbl3vbt2138zTffxF3Pvy1ghQoVslomyj2/jidPnhxIGzBggIsPO+wwF0f8kHPKsjPPPDOw3KpVKxc/88wzgbSBAwe6uH79+lkrE490iYiIIsJOl4iIKCIcXs4A/2HZAFCnTh0XP/7441EXp9zxLxXxH2ANAF9++aWLDzjggMjKRLl37733uvi+++4LpN18880uvu222yIrE0WrevXqgWX/Yfe77757IM2//OzRRx/NWpl4pEtERBQRdrpEREQRkXg358+SSDcWlR49egSW/eGqUjSkmY3bZJW6+ly/fn1gedCgQS72H3QdPqsxD2Xrtmelrk4T+e6771wcHiY+4YQTXBxuo1WrVnVxKTqTvVy00dLi/PPPDyz/5z//cfGCBQtc7D8opYRi1iePdImIiCLCTpeIiCgi7HSJiIgiwjnd8oPzRWUL53TLHrbRCPlPmgKAgw8+2MUvv/yyi9u2bZvqJjinS0RElEvsdImIiCLC4eXyg0NXZQuHl8settGyhcPLREREucROl4iIKCLsdImIiCLCTpeIiCgi7HSJiIgiwk6XiIgoIlFfMkRERFRu8UiXiIgoIux0iYiIIsJOl4iIKCJ53+mKyEgR2Soii5Jcv4qIFIrINhEZHGedpiKidr3LM1rgDBORk2w5d4jISbkuT7pEZKCtm0IRqZHke763n4HRCdZREdkgIndlrrSZl8zns7RLoU22sPu8XUR6xlmno/2MF4rIqRktcITKSP2yjaZRh6Wi0xWRSSKy2e5IoYjMK2EW96lq01CeJ4nIdFuJi0XkrwCgqltUtQDAi0nkW0tVh3t5VheRoSKyUkTWishkL+16EflBRNaJyFIReUhEKha3AREZYD9sJ3mv9bHbmCUiB3mvHyMiY/z3q+p4uz8/JbE/kRCRliIy0f6NFojIWSXM4hVVLVDVDTa/WiLyvIissP8G+iurajMAdyeRb2tV7eeVs4v9GxeKyP9E5EAvTURksIj8bPdjkoi0SrDPi0Rkk/cZ/sBLO1FEForIMhE5z3u9lv2M1vT2pSSfz6wRkdoi8pZtPz+KyIUlzCLQJu0X1bO2fSwXkd5Faao63+7zlGLyXGo/F+/ZPOuJyDu2vamINPVXTrRNm36iiMwVkY0i8pGINIm3YTE/xD+y684Ntde8q18AEJHzReRbW8ffi8hxJXh7oI3a/NqIyGT7+f9FRK4rSstSGz1IRN6335XFnhEsIhVsm14qIutF5CsRqWXTIqvDUtHpWr1sJRao6v7pZGQr5iUA/QDsBuBQANMyUMbhAGoDaGn/v8FLGwugjaruCuAgAK0BXFtMOZsBOAfAMu+1egAuA7AvgGEA7rGvVwTwIIDrM7AfWWPL+TaAd2H+RpcDGC0iLdLI9iEA1QE0BdAOQHcRuSTNcjaHaTRXAqgFU3/veD+UzgVwKYDjYPbjEwCjism2i/cZPtl7/WEAXQCcCuBJEalgXx8C4B5VXZ/OvmTJEwC2AqgLoBtMueP+6EjCQADNATQB0AnAzZL+EesOAO8B6FrSbYrIHgDeBHA7TP1+CeCVBNt6GcBXAOrAfK+8LiJ72rS8q18R6QzgXgCXAKgJ4HgAP6SR3x4wdfEUzN9oPwAfJHxT8XkW10a3AXgV5vsyGXcAOBrAUQB2BdAdwGabFlkdlqZON5P6A3hKVcep6u+qukpVv08nQxHZH8DpAC5X1V9Vdbuquo5cVb9X1TVFq8N8IexXTLaPA7gF5sutSGMAX6nqOgDjYTpfwHS276jqonT2IwIHAKgP4CH7N5oI4GOYD3iqusAcOW20+z8CpkNMxykApqjqVFX9HeYLqAGADjZ9HwBTVfUHVd0OYDSAA2NnVawaqjpLVWfC1HUdEWkHYB9VfTW93cg8MUOGXQHcrqqFqjoVwDtIrw4vAnCnqv6mqt8CeBrAxemUU1V/UdWhAL5IYZtnA5itqq+p6maYDrq1iBwQzsT+YGwDYICqblLVNwB8gz86+7yqX+sOAINU9VNV3aGqP6vqz2nk1xvA+6r6oj0SXG//5ulI2EZVdZ6qjgAwu7iMRGR3mO/Qv6vqj2rMsnUPRFiHpanTHWKHCT4WkY5FL4pIYxFZIyKNS5BXe/veb+xwwWgRqZ1m+Y4E8COAO2w5vxGRwC9sEblQRNYBWAlzpPtUvMxE5FwAW1X1/0JJCwAcbIc9TgIwW0QaATgfwANp7kMUYj3OSmCO/s2Cqc9j08g3kF+KJEaefr7/BrCfmPnGSgB6wPyST+RFEflVRD4Qkdbe6ytEpLV9bQeA32B+WSccCcmhFgC2q+p877WZAFoBJW+T9guvvs1jp/yyIYlttvLT7DDp93HK1ArAD6GjHT+vvKpfexR3OIA9xUz/LBGRx0WkmrdOSdtoewCr7RDwChEZW8Lv7JhFReI2WhIHA/gdwDl2qmG+iFztpUdWh6Wl070F5oiuAcwQ7lg79ApV/UlVa6lqSeYsG8L8Ku8KM7xUDcBjaZaxIUxlr4VpzL0APC8iLYtWUNWX7PByC5ih4V9iZSQiBTDzGzsNFavqKgB3AZgI4M8AbgLwCMzf6CwR+a+IvC0iDdPcn2yZC2AFgD4iUklETob5ZVq9aAVbn1NLkOd7AG4VkZoish/MUW71Yt5TnA8BdBBzgk5lAH0BVPbyXQYzxzgPwCaY4eYbYmVkdYMZ/m4C4CMA7xfNF8EMjz0C89nuDuAfACYAqGrnpD4SkQ47Z5kzBTCfc99amGHIVNpkgZfHTvllSXHbTLiPMfJKtG6+1W9dAJVgpraOg5l+OwxmhBBASm20IcwP0+tgRusWwgzJp6O4NloSDWGmGlvAjGKdA2CgHWYHIqzDUtHpqupndjhii6o+DzMc+ac0stwE4Dl7gkYhTAcXNz8RmS1/nAAT72SCTTBzCINVdauq/hfmy/Xk8Iqq+h3MkMfQOHndAWCUqi6MlaiqL6tqG1U9Daaj3wIzn/QAzFDrayilR72qug3AmTA/GJYDuBFm3mVJGtleC/P3/w5mvvjlRPmJyDivPrvFKedcmC+Jx2E62D0AzPHyHQDgCACNAFSFqbOJIhKzwavqx3bocaOqDgGwBuYLDao6Q1U7quqRdhuXwnwmn7H5XgJglIhk68H0JVUIM+fl2xVAqvNahV4eSeXn1V9hikdMxW2zJPuYcN08rN9N9v/HVHWZqq4E8C+k/537lqp+YYds7wBwtIjsFmvlDLXRkpYPMEPqm1T1a5jRrD/ZbUVWh6Wi041BEXuYMllf2zyS25hqK+8EmHhnUH5dwjJUBNAsTtqJAK61wxzLYb7YXxWRW/yV7HDP3TAdV3MAi+1c7xcADilheSKjql+ragdVraOqp8CMYnyeRn6rVbWbqu6tqq1gPrdx81PV07z6jHuGoaq+rqoHqWodmE62Cf6YH2wNc4bmEjXnBYwEsDuSn9eN9xl+CEB/Vd0EM+T1pZ2nrgRgzxjr58J8ABXtiSxFWiOJubNYVPU3mC9Nf8g9YX5e/RWUcJQr2W3O9tPsPHazOGWaDWBf8c5gTVD+Ul+/9m+zBCX4jkxC+Du3KI75PZ6hNlrS8vnlSiSrdZjzTlfMKdmniEhVEalof/UcD+D9NLJ9DsAlIrKvPTK5BeZs2nRMhrks5zZbzmMAdCwqp4j0FJG9bHwggNtghidiORHmCPZQ+28pgCtgzhj19QcwUlWX2m3vLyJ1Yc7ETPlMw2wTkUNsfVYXkZsA1AMwMo38molIHTGn/J8Gc0Z02tc4ikhbm+eeMPPvY+2va8A07HNFpK6I7CIi3WEa3YIY+TQWczlXZbvffWB+lX8cWq8zgKqqWvRZXAjgBDFnBVcBsCrdfcoEO7/5JoBBIlLDftbPQPFnbyfyAoD+IrK7mJOV/o40PhNFRKQqzN8OAKrY5WS2+RaAg0Skq33PPwF87dW/Y+e2ZwAYYOv3LJgfvW+EypIX9Ws9B+AaEdlL/jjJKJ3vyOdgpr8OFXMOxO0wJyKuKeZ9CSVqo2JUhRlyhq2bKrHyUXMi7RQA/cRcStYSwHkI7XMkdaiqOf0H88vhC5ihmjUAPgXQ2UtvDDO80zjO+0fCDPmGX78DwK/23ygAuyfzPpvWFOYXUcXQ661gLh3ZADMEcZaX9hzMHO4GAIsA3G8rryh9NoBucba3CMBJodf2t3+Xit5rfWBO0poD4ODi8shhnd4PcyJCIYBxAPYLpRcCOC7OewcCGB167a8wP0w2wnz5nZLM+0LpGqMcU+3nbjVMg67hpVWF+RG0DMA6ANMBnOqlDwMwzPtcfG3rfhXMj63DQ9uqYsvexHvtRFtvywCcn+znM6I6rA1gjN2nnwBc6KWVuE3a/X/W/i1/AdA7xvsmAegZJ8+OAJbEqdfAv2S3CXOi4lyYocdJAJrGql+73NSuswlmnj/cXvOtfivBTH+tgZkGehTB76sStVH7+j8A/AzT9scCaJTM+0J1WZI22jRG/S/y0scB6OstN4A5P6QQ5qDlilzUYU4qPMMfnqftH/H7JNevYj9oG2AuAYi1ThOY67fWwJxinvP9TLA/J9pybgLQKdflycD+9Ld1s8ZvYMW8Z579DDybYJ3NMCe/3JnrfSxmX4r9fJb2fym0yeZ2nzcCuDjOOsfbz/gaxPjRlS//ykj9so2mUYd8tB8REVFEcj6nS0REVF6w0yUiIooIO10iIqKIFPsUnAzjBHLuZOPCfNZn7mTrRgus09xhGy1bYtYnj3SJiIgiwk6XiIgoIux0iYiIIsJOl4iIKCLsdImIiCLCTpeIiCgi7HSJiIgiwk6XiIgoIlHfHIOIiMqZ33//3cWdO3cOpE2ePNnFPXr0cPGzzz6b/YLlAI90iYiIIsJOl4iIKCLsdImIiCLCOd0s2Lx5s4tXr16d1Htq164dWB4xYoSL27Rp4+ImTZoE1qtfv34qRSTKG999911guVmzZi7+9ddfXfz+++8H1nvzzTddfPbZZ8fN/6ijjnJx8+bNUy4nBfnzuIMGDXLxlClTAuuJ/PFcgPbt22e/YDnGI10iIqKIsNMlIiKKiKhG+rjFMvNsxxkzZrj4tddeC6SNHTvWxbNnz04qv0MOOSSwPH/+fBf7w9Vh27dvTyp/8FmdZU2Ze57uli1bXHz55Ze7+I033gisV6NGjZjvWbduXUrbrV69uosLCgoCaWPGjHFxBEOfZaqNjho1ysWXXHKJi88555zAekOGDHHxPvvsk/2CRYfP0yUiIsoldrpEREQR4dnLIf7Zxk899ZSL77777sB6mzZtcnEmhui//vrrtPMgymeDBw92sT80GbZx40YX+2f2N2rUKLDebrvtFjePHTt2uHj06NEx8waA0047zcXhqSJeOZDYzz//HPP1P/3pT4HlMjakXCwe6RIREUWEnS4REVFE2OkSERFFhHO6IStXrnRx//79s7qtww47zMVHHHFEVrdFwfn6tWvXBtL8y1LCdzaqUKGCi2+++WYXt27dOrBenTp1MlLO8mL58uWB5aeffjrmek2bNg0sjxs3zsX16tVzcdWqVQPrVa5cOe62/fMw/EuBrr322sB6a9ascfGAAQMCaY899ljcbVOwjfl10alTp1wUp9TgkS4REVFE2OkSERFFpMzekSp86v8zzzzj4o4dO7o4fCeoH374wcXt2rVzcc2aNQPrrV+/3sXnnXdeIO3QQw918dFHH+3i8KnxFSv+MbqfaCgsQ8rU3W7iWbZsWWD5iSeecLH/EIlffvkl7W359QcEpwtOPvnkQNrAgQNd7A9XpyHv70i1cOHCwLL/IAP/JvjhO1KdeeaZWSvTQw89FFi+5ZZbXOzfwB8Apk+f7mK/zachr9uo/50IALVq1XKxf+nVu+++G1WRco13pCIiIsoldrpEREQRYadLREQUkTJ1ydDWrVtdfOqppwbSPv74Yxd//vnncfPYd999XbxgwQIX+/MTQPB0+F133TWQ5s9HUXYsXbrUxf687ZNPPhlYz7/kw9ekSZPA8l/+8hcX+3OLANCnTx8X+5c7jB8/PrCefwnMSy+9FEjzzw/o0qVLzDKVN9u2bYub1rt3bxdncw437IYbbggsDx8+3MXz5s0LpPl1nKE53bx2//3357oIAILn5SQ6d6N58+aB5T322CNrZfLxSJeIiCgi7HSJiIgiktfDy+EHuF9xxRUu9oeTgeClAOHLhOIJDyn7Ej3BhDKvX79+geVnn33WxYmGkM4991wX+0OA/pAxsPPlP74pU6a4eNiwYS6+6KKLAutNnjzZxQ0aNAik+UOk/qUV/gPUy5twHfg6dOgQYUni8z8//lOQAGDixIlRF6dUe/HFF+OmhYft03X77bfH3bZ/V8HwpaO+8Pe7/yS5yy+/PN0ixsUjXSIiooiw0yUiIopI3t2Ryj9DeejQoYG0G2+80cV169YNpPlntJXTm5OX+rvdhO/4499F7Oqrrw5u2Pvc7r333i4OP6SiZ8+eLk71rl/+DfHHjBnjYn8YC0h+2qKwsNDFaQwv5+UdqX777TcXH3PMMYE0f5rgiy++cLF/RUHUPv30Uxf7d5cDgDZt2rj4yy+/zMTmSn0bDfPPQN9vv/0Caf6d1/zv30R27Njh4sWLFwfSjj/+eBcvWbIk7vv874M//elPgfUmTZoUt0z+FQ0zZ850cRpTibwjFRERUS6x0yUiIooIO10iIqKI5N0lQ5988omL/TlcIHgnofAcSzmdx80rc+bMCSz7D4wPn3vQuHFjF//3v/91cfhOU8ny54TWrVsXSLvmmmtc7M9Drlq1Km5+4fJef/31Lq5SpUpKZSwLXnnlFRfPnTs3kPb3v//dxbmcx6Xkvf322y7++eefA2l9+/ZNKg//ErpRo0a5+Nprr437Hr/9h9e97LLLXBy+W6Dv4osvDiyPHj3axatXr3Zxpi8P5ZEuERFRRNjpEhERRSTvhpcnTJgQN80/pTzRsAKVTv4QL5D4LlH+5T/+VMKrr74aWG/27Nkx3x++VMd/IPm0adMCaf7lZ/6DFhKpX79+YNm/o1aGHmKfl/wHCOy+++6BtER3qKLSyb+kKuzAAw9MKg//QQl33XWXi8MPjjnnnHNc/PDDDwfS/MuEknXAAQeU+D2ZwCNdIiKiiLDTJSIiikjeDS8//fTTcdNefvllF7dt2zaQ5j/DtGHDhpkvGKWtVatWgeWzzjrLxeFh4++//97F/rBTomcZ+8O64YdlJBJvSHmXXYK/Wf2zJh988PX2eiEAAB6WSURBVMFAWkFBQdLbKy+OOOKIwHL4jkZU+oXPWE5G+E5u4WdgF7ntttsCywMHDnRxNqZo/IdshM+OziQe6RIREUWEnS4REVFE2OkSERFFJO/mdJcvX+7i8Jza5s2bXdyrV69Amn/HEv9JNJ06dQqst2DBAhe3bNnSxYnukBN+WoU/N8lLl5JXqVKlwPKIESNc/MQTTwTS/GX/ySF77rlnYL2mTZu6eMuWLS72n2IDAOPHjy9xecN33PGXeQc0w38qGBCsA8p/a9ascXH4LmzxnmAXbsv+3Z/8u7/deeedmShiXGvXrg0s+3eKy+ZlfTzSJSIiigg7XSIioojk3fDyvffe6+LwKeWJ+Hc7GjRoUMw4U/y7o5x55pkuDg+rUPLCw7X+wy7CD75Ixg033BBYTjS87N/w/N///reLO3fuHFgvPN1BwYdRAMC3337r4kaNGkVdnBLzH9AQFp4OKY/8S/TCl+vFu3wvfJmRv16yd3xLlf9whfvuuy+Q1rNnz6xuuwi/JYiIiCLCTpeIiCgi7HSJiIgikndzujfddJOLzzvvvEDaX/7yFxeHL1XwL+sJP80m0/zLmp566ikXH3rooYH1/Id2U/a98MILLi7J/Lr/oG7/SVZU9ixevDiw7D9UPSze7QspsUceeSSw/H//938ufvfdd13sXzIIABdeeKGLq1WrltK2u3bt6uLwk8ZuvfXWlPIsKR7pEhERRYSdLhERUUTybnjZvyyjSZMmgbRvvvkm7vv8SxW2bdvmYn+4GgAmTJiQbhED/LuyhB/4zOHl7Hvvvfdc7N+V7Pfff4/7nvDTb44++ujMF4xKDX9I2X+gOhC8W9Kf//znQNohhxyS3YKVQv4lN0DwaV/JCg8Nz50718Wnn366i6+44orAemPHjnVx+FIu/25SX331lYtvueWWwHr+JWyPPfZYIG2fffYptuyZwCNdIiKiiLDTJSIiikjeDS+nyn94gc8/Iw4IDi9XrPjHn6dPnz6B9fyhjwceeCCQ9vjjj6dcTkrPjz/+GFi+4IILXLxu3bq47/MfTPHSSy8F0vzPAZVM+K5T/t29csm/gsG/y93QoUMD6/lTWOHhyPJ4B7KaNWsGltu0aeNi/2ExADBu3DgX+2cNh+/kVVBQ4OKJEye62B8mBoJ1EX5whv+Ae/9OU+EzlP06vPzyy5EL5e9TQ0RElCPsdImIiCLCTpeIiCgi5X6y6sQTT4yb5l9WMmTIkEDa/PnzXfzmm28mta18eKpKvnvnnXcCy+EHVRepUaNGYPmzzz5zcbNmzTJfsHLqgAMOCCz7l2WsXLkykLZx40YXh+fiUuE/sSY8V/vJJ5+4+KOPPoqbx/vvv+/ipk2bpl2mssa/496iRYsCaaNHj3bxqlWrXHz77bcH1ttzzz1j5j116tTAcr9+/Vz8wQcfBNL8SzNbt27t4mHDhgXWa9euXcxtRYlHukRERBFhp0tERBQR8Q/LIxDpxpLh350KCD7cPNUbmleoUMHF3bt3j5tf5cqVU8o/RbGfKJ2eUlGf/uUD/qU/wM71W6Rv376B5cGDB2e+YNmVjfoEslyn/iUmM2bMCKR16NDBxZmYivGHIFesWBF3vbp167r4oosuCqT5n4sIHlqf1220sLAwsHzssce6eNasWUnl4fdH/sPti3PjjTe62L98KNUHI2RIzB3gkS4REVFE2OkSERFFhJ0uERFRRMr9nG7Yhg0bXPyPf/zDxf7tyQBg2bJlLm7RokUg7ZprrnHxVVddlekipiqv54vCtm7d6mL/7//TTz/Ffc9RRx3lYv9pI0Be3uoxL+d0P//8cxeHb606ZcqUrG03fMtG/zIV/3LAiy++OGtlSEKZaqObN2928fjx4108Z86cwHr33HOPi/2nAiWa073kkksCy/EuO8oxzukSERHlEjtdIiKiiHB4OUmTJk0KLPt3sbn55psDaeG7HZUSZWroatq0aS4+/PDDXZxoSMof1grfKSkP5eXwsi/8QPTOnTu72B+GTtVtt93m4vbt2wfSunTpknb+WVCm2ihxeJmIiCin2OkSERFFhMPL5UeZGrryz0T2H1YQ5j/Q+qabbspqmSKW98PLtJMy1UaJw8tEREQ5xU6XiIgoIux0iYiIIsI53fKjTM0X+Q9D9x+eXa9evcB68+fPd3FBQUHWyxUhzumWPWWqjRLndImIiHKKnS4REVFE8u4u70RA8Cb1F1xwgYsff/zxwHplbEiZiPIcj3SJiIgiwk6XiIgoIux0iYiIIsJLhsoPXo5QtvCSobKHbbRs4SVDREREucROl4iIKCJRDy8TERGVWzzSJSIiigg7XSIiooiw0yUiIooIO10iIqKI5H2nKyIDRWSbiBSKSI0k3/O9iGwVkdEJ1lER2SAid2WutJknIi3svm8XkZ65Lk+6RGSkrZtFSa5f7P6LSEcR2WHXOzWjBc6wslafANuoiFSx+75NRAbnujzpYhuVy2w5VUT2K+n7S0WnKyItRWSiiKwVkQUiclYJs3hFVQtUdYPNr5aIPC8iK+y/gf7KqtoMwN1J5NtaVft55ewiIrPsH/x/InKgl1ZFRB4SkaUi8puIDBWRSsVtQER62Mrr6b12oYgsE5GFItLRe72Z3W4Fb1/mq2oBgClJ7E8kRKS2iLxlvxB/FJELS5jFfara1Muviog8KyLrRGS5iPQuSivB/i+1n5H3bJ4iIv1E5Ceb779FZFdvmw1E5G0RWS0iS0TkymL2+RpbX+tE5EsROdZLy+v6BAAR6WX3a4uIjEwhi3Ab7SQiH9k2vyi8cmloo8XkdaKtz2Uicp73ei0RmS4iNb192WLr9MUk9icSIjJJRDbbfSsUkXklzCLcRos64kLvXwUg9TZq891TRF4SkTW2zl700u4TkcW2zf0oIv1iZxskIs+FO0wRedjm/4mINPBe7yYij/jvV9URdn9SkvNOV0QqAngbwLsAagO4HMBoEWmRRrYPAagOoCmAdgC6i8glaZazOUyjuRJALQBjAbxjyw8AtwI4HMBBAFoAaAOgfzF57g7gNgCzvdcqArjHvv8aAP5jcx4F0FtVt6ezLxF4AsBWAHUBdAPwpIi0SiO/gQCaA2gCoBOAmyX9X8MXAegO4BgA9QFUA/CYlz4awEKYffgzgLtFpFOsjETkSJg6OwfAbgBGAHhLRCqUkfoEgKUABgN4NkP5bbB59clQfhlto0nk9TCALgBOhfl8F/1wGgLgHlVdn6n9yqJetpMrUNX9M5DffV5+BRn6XL8JYDlM298LwANe2ggAB6jqrgCOBnChiJydKDP7Y7hZ6LV2ANoC2BvAVJjvZIjIbgBuAvDPDOyHk/NOF8ABMF96D6nqdlWdCOBjmC/EVHWB+QBsVNVFMJVzaZrlPAXAFFWdqqq/A7gXQAMAHbxtPqqqq1X1V5gv1OK2OcSut9J7rQ6An1V1GYDxAPYFABE5x77+aZr7kVVihg+7ArhdVQtVdSqAd5BefV4E4E5V/U1VvwXwNICL0yxqFwAjVHWxqhbC1Od5IlJdRAoAdARwl6puU9WZAF5H/PpsCmC2qk5Tc+H7CwD2gPmSyOv6LKKqb6rqGACrMpTf56o6CsAPmcjPymQbLS6vGqo6y342tgKoY7+891HVVzO4T+WWiJwMoBGAPqq61rbFr4rSVXVe0ciJtQNA3OFe+4PpMQC9Qkn7AJiqqlsATIBtowDuAnC/qq5Nf2/+UBo63Vj3pxSYX6NmwQwtHBtjvWTzDeSXIomRp59vrPSG9tfSzpmZBno4gGGhpF9hGnBDAJ0BzLadQH/YX2ClXAsA21V1vvfaTACtAEBEGtv6bJxMZnY0oL7NY6f80hCrvqrAHFGL95qfHu8zNA5ABRE50h7xXApgBswv9Hyvz6Sk2EYzXgxkro0Wl9cKEWktIq1hvux/gzn6vTbdnYjQEBFZKSIfh6Y9StRGPVeJmY6ZJiJdM1C+9gDmAXheRFaJyBci0sFfQURuFZFCAEsA1ADwUoL8bgAwWVW/Dr0+G8BxIlINwIkwbfRwAPuraqL8UlIaOt25AFYA6CMileyvmw4ww8MAAFWtZY+YkvUegFtFpKYdt7/Uzy9FHwLoIGbCvzKAvgAqe/mOA3CdnYPYG380vp22a7+YhwK4RlV3+Gl2+R8wR1Y3Afg7gEEwv9AOFjMP9r6IpPsjIlsKAIR/Ga4FUBMAVPUnW58/lSC/ojx2yi8N4wD0FJGm9kv3Fvt6dTs0+DGA20Wkqoi0gTl6j/cZWg/gDZihqS0ABgC4XI18r8+kpNBGsyFjbTSJvK7E/7d370FXFPkZx58GUVRENwjiouIdjcaXIlJQhgVXiqWWWIoYhJKFuEIBUYSAlwCKsIJX1NVSEBURopSUCmpQtlIWAgay4EIQshgwpYKAIF6iy7VKpfNHD23PeM57Oe85fd7L91P1Vv3mdM+cOe+cmT7T3dMtPS7pGblanH+Su0tqnhzPZdkCoo75F7k7unZyn2GxMeYcqaBzVHK1BufJ1e5MkjTXGPN3tdzH0yT9StIyuarfRyS9YYw5+UgGa+0DcteCTpJe0E+vPZIkY8zpkkYoR1WxtfbPcufvaklnyNVqPC5ptDFmtDHmXWPMfGPMSbX8PJLqQKFrrf1OUl+5drPdkm6V9LLcL5dCjZZ0UNL/yrUXv1TZ9owxfwga/wfl2c/Nkv5Rrk1ul1z14QfBdu+VtF7uDuc/Jb0u6Tu5HxRZN0naaK39Y573Wmqt7Wqt7SH3K/pSSXPlvlQ3SJoqaXa+z1Nm+yS1zLzWUq5gKnR7R7ZRre2ZdGeOfL/W58h9L5bL/dJdlrx+5HgOkqt22i7pKbn2vXzfoWFyP+wukrsw/0bSm8aYn0v1/njWCbHP0aq2Za1931p7ubW2S/L6jXIdv2ZL+p2k30p6wRhTqtmgasVau8Zauzfp5DVP7kdmn1ps77+stV9Za7+31i6RO1/ytq9W8xw9KGlr0nHpO2vtArnzMVWYJz9u1yf5f5dnW49JuidfVbG19vfW2gpr7QBJA+Q6fTWR62PUU9L/yPUJqLWyF7qSZK3daK3tYa1tZa3tLfcL7L1abO9ra+0ga21ba+1Fcp8z7/astb8OGv/z9jC01r5qrb3YWttK7m6mvaQ/JWkHrbWjrLXtrLVny7V9rcvTmaCnpGuM64m7W64TwCPGmLCTjZIT9km5HxEnS2pqrd2WvOcl1f6HxPWhpKOSjihHVCjoLFYT1tr/k7voVVR3e5nOHDl/rVtrD1trJ1trz7TWnpZsb2fyJ2vtNmvtldba1smFtZXyf4cqJC22rpfmYet6X+6SO65ePT2edUIZztFKt5Xxe0l3WWsPSvobSWut60vSTFLrgj90XFbFnVqw0u1V5xyVtFE1m5rwKGU6SQV6SpoeXHMl6Y8m82SFMeYUuTvie+SaEjYmN4ZFO0frRKFrjLkkqcY7zhhzm6RT5e4ECt3eOcaYVknv0V/L/Vqp9fNxxpi/TbbZWtLTchfazUlaO2PMz43TVa6KZXKeTd0g6UJJHZO/tXK/0LJd3odJWm+tfV/uAnGscY8t/FLF7YBSNEnHhkWS7jHGHJ9UMV0td1dXqH+VdJcx5mfGmAvkqmjn1mY/jXus6ZzkeP21pEflfgkfTtIvTJonjjbG/EaumuvRPJv7k6S/N8acnWyvl1zb9p8z+erd8TzCGHOUMaa5pKZy7dfNzY89eQvZXpNke83commeVOPWdj+LdY5Wuq0gTy9Jza21byYvfSLpCuN66x+jInU8KybjHmvqfeQYJjUH3SX9ey22+Q/GmBbJcf2VXG3Pv9VyV1+T9DPjHqtsalznw3aSViXvMyK5Jhjj+sjcLFfFn8v5cj+Oj1xzJdex7rVMvkclTbbWHpA7lp3Njx0ri3OOWmvL/idpulxHhH1y7S7nZtL3SfpFnnWnSHox89p1co84HJCrSupdnfUy6TbHfqyUq9b8Wu4kPD5I6y5pa/KeWyQNyqz7B0kT87zXcknDMq+dLHfRbhm8NkiuCn6rpF9WtY0yHs+/kqu62y/pU0nXB2lnJMfzjDzrzpU0LfPaMXLVwX+R9LncYzZV/g+DtMsl7ci8dn5ynA5I2pbdpqR/lusEtT857pfm+07K/aK/J/mse+WqogY3lOOZ7M+U5JwI/6bk+n/kWTd7jl6eY3vLq1ovk17Sc7SybQXfy/cltQ9e65m8xy5JA6v6bpfpWLaW+6G4V9I3cm2ZvYL0Qs7R/5BrT/2LXEfHgTnWy/udVo5zNHn9F5L+O9mftcE510Su787XSdqHcu3upprfyVzfnV9Keivz2mNyZdNqSadVtY1q/f/L/QUowhfoLrkL4zfZk6KSdbYkB2ROJXkOJV+iqeX+jFV8lvOSz35A0g3l3p8ifJ5nk2PzUbE+f3KxPZjk+8kPsLr019COZ/KZGvs5ekzy2ffL3UWVfZ9q+Xka+zn622Q/D0k6u6brM58uAACR1Ik2XQAAGgMKXQAAIim492GBqMsun1I8L8jxLJ9SPf/JMS0fztGGJefx5E4XAIBIKHQBAIiEQhcAgEgodAEAiIRCFwCASCh0AQCIhEIXAIBIKHQBAIiEQhcAgEgodAEAiIRCFwCASCh0AQCIhEIXAIBIYs8yBABFM3PmTB/ffPPNqbR+/fr5eOHChdH2CTWzfft2H7/22ms+njt3birfhg0bfGxtevIkY36c0KdHjx4+rqioSOWbMGGCj9u0aVPYDtcSd7oAAERCoQsAQCRUL6NBO3TokI+3bt3q48WLF6fy3XHHHT5u0iT9W3TEiBE+Puuss3x80003pfIdf/zxtdpX1NzSpUvzpi1atMjHW7ZsSaV16NChZPvUWC1ZsiS1HFYHv/3223nXW7FihY/DauLKqpAnTpyYShswYICPL7744mrucXlwpwsAQCQUugAARGKyt/AlFvXNkGKqzlJjdeJ4/vDDDz5+4YUXUmlTpkzx8Y4dO/JuIzwPwmqsyowdOza1PH369GqtVySlOJ5SHTmm1VXdYzVjxozUcrZpoI6o1+dotlkmPDbHHXecjzt37pzK16tXLx+HvY1btWqVytelS5ei7GdEOY8nd7oAAERCoQsAQCQUugAAREKbbkbY7hd2S58/f34q3+HDh31c2SMm06ZN83GLFi1S+TZt2uTjbDf3Zs2a1WS3q6NetxdVJnw05LrrritoG3379vXx66+/XtA2vv/++4LWKxBtuqp+m27k61yh6vU5Onz48NTynDlzfNy9e3cfv/POO7F2qdxo0wUAoJwodAEAiKRRVi+Hj5hkR6rp06ePj4vxiMmYMWN8/Omnn6bSwmrRbJVLOGh3kdTrqqus3bt3+7hTp04+3rNnT7XWnzdvXmp54MCBPn788cdTaeFoVZWhejk+qperFO2DHzhwILXcsWNHH+/du9fHmzdvTuU78cQTS7tj5UP1MgAA5UShCwBAJBS6AABE0ijbdMPZZs4999y8+U4//XQfv/LKK6m0Y489Nu964aTM4fBn/fv3T+UL2znWrVuXSrvwwgvzbr9A9bq9KGzDlaTx48f7OBz6MdvGd9555/l42bJlPj7llFNS+cL1wjZ/Kd22f9lll/n4888/T+W79NJLfbx69eocn6KoaNNV5ZPYhxgGMr7nnnvOx+HjROH1V0pfZxsY2nQBACgnCl0AACJpNJPYh9WTYRVhVlgF/NBDD/m4JlUgZ5xxho+vvvpqH3/11VepfOH2S1Cd3KBkHzMIq5TD0cGOPvroVL5Jkyb5uG3bttV6r6ZNm6aW27dv7+Nx48b5OKzilqQ1a9b4eMKECam0+++/v1rvjZqprEoZ5RU2XYbxJ598UtD2wvO3BCP2RcOdLgAAkVDoAgAQSaOpXp46daqPw16ngwcPTuV75JFHfJydRLm6tm3b5uN33303b74rr7yyoO03RgsXLkwth72Nwwkn+vXrl8p3/fXXF3U/br31Vh+/+uqrqbT33nvPx6tWrSrq+wJ1XXZEqrD5LDxfr7jiilS+fKP7ZZ+sGTp0aM5Yql8T3HOnCwBAJBS6AABEQqELAEAkDbZNN/s4x6xZs3x8wgkn+PjBBx9M5SukHTc7gtHkyZN9HLZLXHPNNal8559/fo3fqzE5ePCgj5cuXVqtdUaOHFmq3fmJUaNGpZaHDBkS7b2BuiBsx+3WrVsq7aOPPvJxOGtaRUVFKl+vXr1ybvvhhx9OLS9fvtzHs2fPTqWFbcHhiFfZxwTrwqNG3OkCABAJhS4AAJE02OrllStXppbD6oeWLVv6ODvwfXWFVcqPPfZYKu2NN97I+b733ntvQe/VWIXVUx9++GHefFdddZWP68qjA9kJGvbt2+fjFi1axN4doCR27tzp4w0bNqTShg0b5uOnn366xtvu06dPajmsyl68eHEqbcGCBT4+88wzfdypU6dUvrfeesvHbdq0qfE+FQN3ugAAREKhCwBAJA22ernYvvnmm9Tys88+6+Ps4PahcKKEcOB8VC2cQKAyYQ/05s2bl2p3auTjjz9OLYejlF100UWxd6fRq6Pz59Z74XzV2ac4ii2cm3zAgAGptHB5yZIlPs72gA57TocjyEnx5vXlThcAgEgodAEAiIRCFwCASBpsm262q/jq1at9HM4ylB1FJZ8dO3aklrdv3+7j8LGgrHDWm7rS3lhf7N+/38fZGUdCdWVkr8OHD/s4nPkIQDzho0aXX355Kq179+4+7tq1aypt0aJFPi7lo4dcGQAAiIRCFwCASBps9XI4Gb0kffvttz5+8cUXfRxWO9fEunXrfPzMM8+k0sLl7KD4qL7w2FRWhV9XhFXK9WF/gYYufMxIkl566SUfjxgxIpXWt29fH4ejDGYfT6ot7nQBAIiEQhcAgEgodAEAiKTBtulmJyueN2+ejydOnOjjtWvX5t1GOFxfx44dU2nhRPXZGTQuueQSHxc6ixHqt3Amq1zLAOILh6188803U2nh40SDBg3yMW26AADUUxS6AABE0mCrlyvToUOHnHFNTJs2zcfZx0PCUVCyXdbRcDzxxBN505566qnUcqwZTBqbcMS3cEShrJkzZ6aWmXUI2Wtz7969fbx+/fqSvS93ugAAREKhCwBAJI2yerkQ2UnsQ9meqePGjSv17jQK06dP9/GKFStSaeGkFbfffnvOdUrts88+Sy23bdvWx9dee220/QBQc3v27EktP//88z7OTphTTNzpAgAQCYUuAACRUOgCABAJbbrVdN999+VNGzx4cGqZx0OKo127dj6eP39+Kq1nz54+njVrlo9Hjx6dylfsYzF+/Hgf79y5M5U2cuRIHx91FKdWqWzZssXHlT0mhLgWLFiQWj711FN93KNHj9i7k1M429ydd96ZStu7d6+PlyxZUrJ94E4XAIBIKHQBAIiEOrBK7N6928ezZ8/Omy8cFQelUVFRkVoeNWqUj5988kkfr1q1KpVv4MCBtX7vzZs3+zh8rCCsPpOku+++u9bvhaqFE5agvNasWePjcJIAKX2cil29fODAgdTy8uXLc+bLNkstW7bMx2HzlSQtXbrUx61bt67lHubHnS4AAJFQ6AIAEAmFLgAAkRhrbcz3i/pmtRW2E4SPqEjpmYVWrlyZSuvatWtJ96tApuosNVa24xkOy9mlSxcff/HFF6l8kyZN8vHYsWPzbu/LL7/08QcffJBKCyexDrf/wAMPpPLddtttVe12MZXieEp18BwNHxGSpAsuuKBa64Vt8YXOJhZZvTtHwzbd7HWvSZMf7+natGnj46FDh6byhWXQpk2bfJxtV50zZ07OdaT09ThMyw7nGA7POmbMmFRaCWaEy3k8udMFACASCl0AACLhkaFK7Nq1y8fZieo7d+7s4zpandygnXTSST5et26dj2+55ZZUvqlTp/p44cKFPg5HlpKk4cOH+zg7+0hoyJAhPr7xxhtrsMcohRkzZviYienjC5t2Nm7cmEp7+eWXc64TVhNL6Uczw+rg7DU3fAQprK6WpL59++Z8r3DmL0lq1qxZznwxcacLAEAkFLoAAERC7+VKdOvWzcerV69Opc2bN8/H2ZFY6qh61zOyEIcOHUoth5PdhyNGZUeqyfZEDvXv39/H4QQKYe/MMmg0vZcbkUZxjjYi9F4GAKCcKHQBAIiEQhcAgEh4ZKgSYff1bJsu6qbmzZunltu3b+/jsB0+jAEgFu50AQCIhEIXAIBIqF6uRDg5fTh4upQekQoAgOrgThcAgEgodAEAiIRCFwCASBgGsvFgiLmGhWEgGx7O0YaFYSABACgnCl0AACKJXb0MAECjxZ0uAACRUOgCABAJhS4AAJFQ6AIAEAmFLgAAkVDoAgAQCYUuAACRUOgCABAJhS4AAJFQ6AIAEAmFLgAAkVDoAgAQCYUuAACRUOgCABAJhS4AAJFQ6AIAEAmFLgAAkVDoAgAQCYUuAACRUOgCABAJhS4AAJFQ6AIAEAmFLgAAkfw/jfIPLvAUZ5AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x576 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    ckpt = tf.train.get_checkpoint_state('./data/')\n",
    "    \n",
    "    # 如果模型存在相应目录下，就restore恢复模型的变量\n",
    "    if ckpt and ckpt.model_checkpoint_path:\n",
    "        saver.restore(sess, ckpt.model_checkpoint_path)\n",
    "        final_pred, acc = sess.run([pred, accuracy],\n",
    "                                   feed_dict={\n",
    "                                       x: mnist.test.images[:16],  # 测试集前16个数据\n",
    "                                       y: mnist.test.labels[:16]\n",
    "                                   })\n",
    "        orders = np.argsort(final_pred)  # 新数组为排序后的原数组元素下标，换句话说，新数组最后一个值就是预测结果\n",
    "        print(\"\\nthe test accuarcy is:\", acc)\n",
    "        \n",
    "        plt.figure(figsize=(8, 8))\n",
    "        for idx in range(16):\n",
    "            order = orders[idx, :][-1]  # 预测结果取数组最后一个值\n",
    "            prob = final_pred[idx, :][order]  # 根据下标拿相应的预测率\n",
    "            plt.subplot(4, 4, idx + 1)\n",
    "            plt.axis('off')  # 不显示坐标尺寸\n",
    "            plt.title('{}: [{}]-[{:.1f}%]'.format(mnist.test.labels[idx],\n",
    "                                                  order, prob * 100))\n",
    "            plt.imshow(mnist.test.images[idx].reshape((28, 28)), cmap='Greys', interpolation='nearest')\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**对模型结构的理解**\n",
    "\n",
    "    该模型为单隐层网络，由输入层、一个隐层、输出层组成。输入层有784个神经元，隐层有100个神经元，输出层有10个神经元\n",
    "**对模型训练过程（梯度如何计算，参数如何更新）的理解**\n",
    "\n",
    "    采用随机梯度下降SGD的优化算法，损失函数采用交叉熵损失。\n",
    "**对计算图的理解** \n",
    "\n",
    "    1. 先对计算图各部分进行定义，定义输入数据x，定义权重(矩阵)W，定义偏置b，定义运算规则logits，定义模型输出y，定义损失cross_entropy，定义优化器。定义session。对应是计算图的node。\n",
    "    2. node包括name，op，attr信息。通过这些信息能生成计算图。\n",
    "    3. 使用计算图前需要先初始化参数，然后再运行。\n",
    "**解释这的模型为什么效果较差**\n",
    "     \n",
    "    1.学习率为固定值0.3，可能需要采用衰减形式\n",
    "    2.step太少了，整个训练才用了1000*32=32000个样本，1个epoch都没达到\n",
    "    3.隐层神经元个数少，可扩大单隐层神经元个数或扩大网络层数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 三、优化神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、计算图\n",
    "#### 1、网络结构及初始化\n",
    "\n",
    "     网络：双隐层网络结构\n",
    "     输入层：每条数据为1*784的1维数组，一次输入1batch数据\n",
    "     隐层1：共256个神经元；w：784*256，高斯初始化；b：1*256,非零初始化\n",
    "     （激活函数：Relu）\n",
    "     隐层2：共128个神经元；w：256*128，高斯初始化；b：1*128,非零初始化\n",
    "     （激活函数：Relu）\n",
    "     输出层：共10个神经元；w：128*10，高斯初始化；b：1*10,非零初始化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-2-77dbbe2d960e>:2: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please write your own downloading logic.\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ./data/train-images-idx3-ubyte.gz\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.data to implement this functionality.\n",
      "Extracting ./data/train-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use tf.one_hot on tensors.\n",
      "Extracting ./data/t10k-images-idx3-ubyte.gz\n",
      "Extracting ./data/t10k-labels-idx1-ubyte.gz\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\contrib\\learn\\python\\learn\\datasets\\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n",
      "WARNING:tensorflow:From D:\\Anaconda3\\lib\\site-packages\\tensorflow\\python\\framework\\op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "#导入数据\n",
    "mnist_new = input_data.read_data_sets(\"./data/\",one_hot=True)\n",
    "\n",
    "# 定义输入和输出的占位符\n",
    "x = tf.placeholder(\"float\", [None, 784])\n",
    "y = tf.placeholder(\"int64\", [None, 10])\n",
    "\n",
    "\n",
    "# 神经网络结构参数\n",
    "L1_units_count = 256\n",
    "L2_units_count = 128\n",
    "L3_units_count = 10 \n",
    "\n",
    "\n",
    "# 优化方法参数\n",
    "learning_rate_base = 0.3  # 基础学习率\n",
    "learning_rate_decay = 0.96  # 学习率的衰减率\n",
    "regularization_rate = 0.0005  # 正则化项在损失函数中的系数\n",
    "\n",
    "\n",
    "# 训练参数\n",
    "batch_size = 100\n",
    "epoch_size = 60\n",
    "\n",
    "\n",
    "# 存储训练轮数，设置为不可训练\n",
    "global_step = tf.Variable(0, trainable=False)\n",
    "\n",
    "\n",
    "# 初始化：w：高斯分布；b：0.1\n",
    "def initial_weight(shape):\n",
    "    initial = tf.truncated_normal(shape, stddev=0.1)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "def initial_bias(shape):\n",
    "    initial = tf.constant(0.1, shape=shape)\n",
    "    return tf.Variable(initial)\n",
    "\n",
    "\n",
    "\n",
    "# 隐层1：256个神经元\n",
    "w_1 = tf.Variable(initial_weight([784, L1_units_count]))\n",
    "b_1 = tf.Variable(initial_bias([L1_units_count]))\n",
    "\n",
    "logits_1 = tf.matmul(x, w_1) + b_1\n",
    "output_1 = tf.nn.relu(logits_1)  # ReLU激活函数\n",
    "\n",
    "\n",
    "# 隐层2：64个神经元\n",
    "w_2 = tf.Variable(initial_weight([L1_units_count, L2_units_count]))\n",
    "b_2 = tf.Variable(initial_bias([L2_units_count]))\n",
    "\n",
    "logits_2 = tf.matmul(output_1, w_2) + b_2\n",
    "output_2 = tf.nn.relu(logits_2)  # ReLU激活函数\n",
    "\n",
    "\n",
    "# 输出层：10个神经元\n",
    "w_3 = tf.Variable(initial_weight([L2_units_count, L3_units_count]))\n",
    "b_3 = tf.Variable(initial_bias([L3_units_count]))\n",
    "\n",
    "logits_3 = tf.matmul(output_2, w_3) + b_3  \n",
    "logits_new = logits_3\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置正则化方法\n",
    "regularizer = tf.contrib.layers.l2_regularizer(regularization_rate)  # 定义L2正则化损失函数\n",
    "regularization = regularizer(w_1) + regularizer(w_2) + regularizer(w_3)  # 计算模型的正则化损失\n",
    "\n",
    "\n",
    "# 设置指数衰减法\n",
    "learning_rate = tf.train.exponential_decay(learning_rate_base, \n",
    "                                           global_step, \n",
    "                                           mnist_new.train.num_examples / batch_size,\n",
    "                                           learning_rate_decay)\n",
    "\n",
    "# softmax激活的交叉熵计算\n",
    "cross_entropy_loss = tf.reduce_mean(\n",
    "    tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits_new, labels=y))\n",
    "\n",
    "# 总损失等于交叉熵损失和正则化损失的和\n",
    "loss_sum = cross_entropy_loss + regularization\n",
    "\n",
    "\n",
    "# 随机梯度下降优化器\n",
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss_sum, global_step=global_step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "##预测\n",
    "pred_new = tf.nn.softmax(logits_new)\n",
    "correct_pred_new = tf.equal(tf.argmax(pred_new, 1), tf.argmax(y, 1)) \n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred_new, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "after 549 training steps, the loss is 0.574996, the validation accuracy is 0.9652\n",
      "after 549 training steps, the loss is 0.356379, the validation accuracy is 0.9724\n",
      "after 549 training steps, the loss is 0.211385, the validation accuracy is 0.9802\n",
      "after 549 training steps, the loss is 0.180185, the validation accuracy is 0.98\n",
      "after 549 training steps, the loss is 0.146387, the validation accuracy is 0.9818\n",
      "after 549 training steps, the loss is 0.118787, the validation accuracy is 0.9832\n",
      "after 549 training steps, the loss is 0.105974, the validation accuracy is 0.9834\n",
      "after 549 training steps, the loss is 0.101997, the validation accuracy is 0.9836\n",
      "after 549 training steps, the loss is 0.112418, the validation accuracy is 0.984\n",
      "after 549 training steps, the loss is 0.0961108, the validation accuracy is 0.9846\n",
      "after 549 training steps, the loss is 0.083815, the validation accuracy is 0.984\n",
      "after 549 training steps, the loss is 0.0875507, the validation accuracy is 0.9834\n",
      "after 549 training steps, the loss is 0.0730739, the validation accuracy is 0.984\n",
      "after 549 training steps, the loss is 0.0825343, the validation accuracy is 0.9844\n",
      "after 549 training steps, the loss is 0.0712577, the validation accuracy is 0.9838\n",
      "\n",
      "the training is finish!\n",
      "\n",
      "the test accuarcy is: 0.9818\n"
     ]
    }
   ],
   "source": [
    "saver = tf.train.Saver()\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "\n",
    "    # 定义验证集与测试集\n",
    "    validate_data = {x: mnist_new.validation.images, y: mnist_new.validation.labels}\n",
    "    test_data = {x: mnist_new.test.images, y: mnist_new.test.labels}\n",
    "\n",
    "\n",
    "    \n",
    "    # 训练模型\n",
    "    for epoch in range(epoch_size):\n",
    "        total_batch = int(mnist_new.train.num_examples/batch_size)\n",
    "        for i in range(total_batch):\n",
    "            batch_x, batch_y = mnist_new.train.next_batch(batch_size)\n",
    "            _, loss = sess.run([optimizer, loss_sum],\n",
    "                               feed_dict={\n",
    "                                   x: batch_x,\n",
    "                                   y: batch_y\n",
    "                               })\n",
    "\n",
    "        # 每训练1000步，打印一次损失值与验证准确率\n",
    "        if epoch % 4 == 0:\n",
    "            validate_accuracy = sess.run(accuracy, feed_dict=validate_data)\n",
    "            print(\"after %d training steps, the loss is %g, the validation accuracy is %g\" % (i, loss, validate_accuracy))\n",
    "            saver.save(sess, './MNIST_model_new/model.ckpt', global_step=i)\n",
    "\n",
    "    print(\"\\nthe training is finish!\\n\")\n",
    "    \n",
    "    # 最终的测试准确率\n",
    "    accy = sess.run(accuracy, feed_dict=test_data)\n",
    "    print(\"the test accuarcy is:\", accy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**如何修改隐层数量，修改后会起到什么样的效果**\n",
    "\n",
    "    训练速度明显比单隐层时慢，但效果比较好\n",
    "**如何神经元个数，起到了什么样的效果** \n",
    "\n",
    "    将原来隐层100个神经元变为256个，正确率变大\n",
    "**如何在模型中添加L1/L2正则化，正则化起什么作用**\n",
    "   \n",
    "    正则化会防止过拟合的作用\n",
    "  \n",
    "**不同的初始化式对模型有什么影响**\n",
    "    \n",
    "    高斯初始化明显比常值初始化好\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
